Automatic Resource Management in Scala

BY TERRY MOSCHOU 

The Beat the News team at Data to Decisions CRC (D2D CRC) have our heads buried (for the most part) in Spark for big data analytics. Naturally, we dabble a lot in the Scala programming language to support our Spark processing. We are always looking at ways to reduce boilerplate code and streamline our work. In this article I focus on how we did this with better automatic resource management in Scala. 

Java 7 introduced the concept of better resource management with try-with-resource constructs (See Java 7 Spec 14.20.3). Scala has no equivalent native language support for such constructs for managing AutoClosable however this is not a bad thing. As Martin Odersky would put it "As Scala programmers we should rejoice that we have powerful library based mechanisms that let us write these things ourselves, instead of overloading the language with cruft like this". So, lets get busy!

We can create our own construct for automatic resource management (ARM) with Scala For-Comprehensions and implicit classes. Example usage of the Scala constructs for implicitly managing multiple resources by importing AutoManageResources' members follows.

import AutoManageResources._
 
val result = for {
  socket <- new Socket(InetAddress.getLoopbackAddress, 8080)
  out <- new PrintWriter(socket.getOutputStream, true)
  in <- new BufferedReader(new InputStreamReader(socket.getInputStream))
} yield {
  val line = in.readLine
  out.println(line.toUpperCase)
  line
}

Compare this with Java's

String result = null;
try (
  final Socket socket = new Socket(InetAddress.getLoopbackAddress(), 8080);
  final PrintWriter out = new PrintWriter(socket.getOutputStream(), true);
  final BufferedReader in = new BufferedReader(new InputStreamReader(socket.getInputStream()))
) {
  final String line = in.readLine();
  out.println(line.toUpperCase());
  result = line;
}

We can see, we have pretty much identical code/syntax, with the advantage that in Scala we can optionally yield/assign a result from the expression, ensuring that all resources are closed in reverse declaration order before returning the result.

Whats wrong with with a simple try/finally in Scala?

Trying to guard against all failure conditions gets messy very quickly. Though the following example seems adequate, it does not handle exceptions correctly.

val socket: Socket = null
val out: PrintWriter = null
val in: BufferedReader = null
try {
  socket = new Socket(InetAddress.getLoopbackAddress, 8080)
  out = new PrintWriter(socket.getOutputStream, true)
  in = new BufferedReader(new InputStreamReader(socket.getInputStream))
 
  doStuff(in, out)
} finally {
  if (in != null) in.close 
  if (out != null) out.close
  if (socket != null) socket.close
}

What if out.close threw an exception? Would socket be closed? No, and unfortunately this is a possibility for AutoCloseable as permitted in their Javadoc. Likewise, what if the try-body threw exception A and the finally clause also subsequently threw exception B. Which exception should be thrown to the outer block and which exception should be suppressed? Currently as the example stands B would be thrown and A would cease to exist and can make debugging A (and B - especially if as a result of A) difficult. For a further discussion and the complications of correct exception handling of resources see Oracle's tech article on Better Resource Management with try-with-resources statement.

What do I need to code to get working right away?

Provide this object and class below.  Alternatively you can use our ARM4S library (see titled section below) which extends the concepts developed here. 

object AutoManageResources {
  implicit class ImplicitManagedResource[A <: AutoCloseable](resource: => A)
    extends ManagedResource(resource)
}
 
class ManagedResource[A <: AutoCloseable](r: => A) {
  def map[B](f: (A) => B): B = {
    val resource = r
    var currentException: Throwable = null
    try {
      f(resource)
    } catch {
      case e: Throwable =>
        currentException = e
        throw e
    } finally {
      if (resource != null) {
        if (currentException != null) {
          try {
            resource.close()
          } catch {
            case e: Throwable =>
              currentException.addSuppressed(e)
          }
        } else {
          resource.close()
        }
      }
    }
  }
  def flatMap[B](f: (A) => B): B = map(f)
  def foreach[U](f: (A) => U): Unit = map(f)
}

The above implementation is an equivalent translation to Scala of Java's syntactic sugar for try-with-resource statements. Such boilerplate need only be defined once and eliminates the need of the developer to ensure resources do not leak under erroneous conditions and that correct exceptions are propagated.

How does this work?

An implicit class included by the (optionally scoped) import of AutoManageResources' members decorates AutoClosables with the methods foreachmap and flatMap and so that they may be used in Scala For Comprehensions (See the Scala Spec on For Comprehensions). Should a resource already implement any of the necessary above traits, the resource could alternatively be managed explicitly by wrapping in a ManagedResource instance as such

for (resource <- new ManagedResource(...)) [yield] { ... }

Scala will rewrite the for-comprehensions using foreach if the construct does not yield a result, or map (and flatmap for more than one resource) if yielding. The above example would be rewritten by the Scala compiler to the following monadic style.

val result = new Socket(InetAddress.getLoopbackAddress, 8080) flatMap { case socket =>
  new PrintWriter(socket.getOutputStream, true) flatMap { case out =>
    new BufferedReader(new InputStreamReader(socket.getInputStream)) map { case in =>
      val line = in.readLine
      out.println(line.toUpperCase)
      line
    }
  }
}

For comprehensions are notably used by sequences and iterables, but they don't have to be - any object or monad that implements foreachmap and flatMap may be used. For example Scala's Option.

Why this way?

Many examples I have seen that attempt to do proper ARM in Scala either 

  • Have different exception semantics that differ from Java's try-with-resources. In particular semantics should abide by the following:
    • The close method of AutoCloseable must be called even if the body throws any Throwable exception including fatal ones such as OutOfMemoryError. Though you should not try to handle such unchecked errors, finally logic should still (attempted to) be executed regardless.
    • The close method of AutoCloseable may throw any Throwable too. 
    • Importantly, any Throwable thrown by close must not mask (suppress) any exception thrown firstly by the body, if any. Instead it should be caught and recorded as a suppressed exception against the original (currently throwing) exception.
  • Do not support the for-comprehension syntax as an alternative for the more verbose (but also supported) monadic style. For comprehensions provide a succinct way of 'stacking' multiple resources to manage concurrently without needing to nest multiple management constructs.
  • Do not provide the choice of implicit management in addition to explicit management.

ARM4S Library

ARM4S extends the concepts developed here for automatic resource management to support any concept of a resource that may not necessarily be an AutoClosable; for example a java.util.concurent.ExecutorService

Usage instructions for inclusion of the free library in your SBT or Maven project can be found on project homepage at https://github.com/tmoschou/arm4s. JAR artifacts, including source code and Scaladoc, are available on maven central.