Tuesday, October 11, 2011

Jackson 1.9 new feature overview

Jackson 1.9 was just released. As usual, it can downloaded from the Download page, and detailed release information can be found from 1.9 release page.

Let's have a look into contents of this release.

1. Overview

One of focus areas on this release was once again to tackle oldest significant issues and improvement ideas; and two of major new features are long-standing issues (ability to inline/unwrap JSON values; unify annotation handling for getters/setters/fields). Another big goal was to improve ergonomics: to simplify configuration, shorten commonly used usage patterns and so on. And finally there was also intent to try to "2.0 proof" things, by trying to figure out things that need to be deprecated to allow removal of obsolete methods as well as indicate cases where improved functionality is available.

2. Major features

(note: classification of features into major, medium and minor categories is not exact science, and different users might consider different things more important than others -- here we simply use categorization that the release page uses)

Major features included in 1.9 are:

  • Allow inlining/unwrapping of child objects using @JsonUnwrapped
  • Rewrite property introspection part of framework to combine getter/setter/field annotations
  • Allow injection of values during deserialization
  • Support for 'external type id' by adding @JsonTypeInfo.As.EXTERNAL_PROPERTY
  • Allow registering instantiators (ValueInstantiator) for types

2.1 @JsonUnwrapped

Ability to map JSON like

  {
    "name" : "home",
    "latitude" : 127,
    "longitude" : 345
  }

to classes defined as:

  class Place {
    public String name;

@JsonUnwrapped public Location location; }
class Location { public int latitude, longitude; }

has been on many users' wish list for a while now; and with addition of @JsonUnwrapped (used as shown above) this simple structural transformation can now be achieved without custom handling

2.2 "Unified" properties, merging ("sharing") of annotations of getters/setters/fields

Another long-standing issue has been that of isolation between annotations used by getters, setters and fields. Basically annotation added to a getter was only ever used for serialization, and would never have any effect on deserialization; similarly setter never affected deserialization. While this is not a problem for many annotation use cases, it would make following use case work quite different from what users intuitively expect:

  class Point {
@JsonProperty("width")
public int getW();
public void setW(int w); // must be separately renamed
}

which would actually lead to there being two separate properties: "width" that is written out during serialization; and "w" that is expected to be received when deserializing. Many users would intuitively expect annotation to be "shared" between two parts of logically related accessors. Same issue also affects annotations like @JsonIgnore and @JsonTypeInfo, requiring use of seemingly redundant annotations.

Jackson 1.9 solves this by adding new internal representation of logical property, and merging resulting annotations using expected priorities (meaning that annotations on a getter have precedence over setter when serializing, and vice versa).

There are also other more subtle changes, related to these changes. For example, class like:

  class ValueBean {
    private int value;

    public int getValue() { return value; }
  }

can now be deserialized succesfully, even without field "value" being visible or annotated: since it is joined with getter ("getValue()"), and getter is explicitly annotated, field is included as the accessor to use for assigning value for the property.

The last important benefit of this feature is that now handling of Jackson and JAXB annotations is much more similar, which should make JAXB annotations works better as a result (code was simplified significantly) -- this because JAXB had always considered annotations to be shared in this way.

2.3 Value Injection for Deserialization

Value injection here means ability to insert ("inject") values into POJOs outside of general data binding: that is, values that do not come from JSON input. Instead, values to inject are specified during configuration of ObjectMapper or ObjectReader used for data binding.

Why is this needed? Some Java types require additional context information to be able to construct POJO instances, for example. And in other cases, you may want to pre-populate values of some fields; and while there are other mechanims (for example, you can pass an existing POJO instance for "updateValue()") method) they are quite limited.

Only two things are needed for value injection:

  1. Means to indicate properties for which values are to be injected, and
  2. Definition of values to inject

Default mechanism is to handle first part by using new annotation, @JacksonInject, so that we could have:

  public class InjectableBean
  {
    @JacksonInject("seq") private int sequenceNumber;
    public String name;
  }

and second part is handled by allowing configuration of ObjectMapper or ObjectWriter instance with InjectableValues, object that can find values to inject given value id. Value ids can be specified as either Strings, or as Classes; if Class is used, Class.getName() is used to get actual String id to use. For above POJO, we could handle deserialization as follows:

  ObjectMapper mapper = new ObjectMapper();
  Integer sequenceNumber = SequenceGenerator.next(); // or whatever
  InjectableValues inject = new InjectableValues.Std()
   .addValue("seq", id)
  final String json = "{\"name\":\"Lucifer\"}";
  InjectableBean value = mapper.reader(InjectableBean.class).withInjectableValues(inject).readValue(json);

For more on this feature, check out FasterXML Wiki's entry on Value Injection.

2.4 External Type Id

Jackson has had support for full polymorphic type handling since 1.5, allowing configuration of both type identifier in use (usually either a class name, or logical type name) and type inclusion mechanism (as property, as wrapper array, as single-element wrapper object).
This covers wide range of usage scenarios, but there is one inclusion mechanism that is sometimes used but could not be supported by Jackson: that of using "external type identifier". This style of type inclusion is used by some data formats, most notably geoJSON.

By external type identifier we mean case such as this:

 {
  "type" : "rectangle",
  "shape" :  {
   "width": 20.0,
   "height" : 40.0
  }
 }

where type is included as a property ("type") that is outside of JSON Object being typed.

With 1.9 we can support such use case by using @JsonTypeInfo with a new inclusion value:

  public class ShapeContainer
  {
    @JsonTypeInfo(use=Id.NAME, include=As.EXTERNAL_PROPERTY, property="type")
    public Shape shape;    
  }
 
static class Shape { }
@JsonTypeName("rectangle") // or rely on class name, Rectangle static class Rectangle extends Shape { public double width, height; }

One thing to note here is that this inclusion mechanism should only be used with properties; annotating classes with @JsonTypeInfo that indicates external type identifiers can cause conflicts.

2.5 Value instantiators

And last but not least, 1.9 also allows much more control over mechanism used to create actual POJO value instances. While Jackson 1.2 added support for @JsonCreator annotation, there has not been a way to add custom creator objects.

With 1.9, we get following pieces:

  • ValueInstantiator (abstract class), extended by objects used to create value instances
  • ValueInstantiators (interface), provider for per-type ValueInstantor instances (as well as ValueInstantiators.Base abstract class for actual implementations)
  • Module.setupContext method addValueInstantiators(); as well as SimpleModule method addValueInstantiator(), for adding provider(s), so modules can easily provide instantiators for types they support
  • @JsonValueInstantiator annotation that can be used as an alternative to specify instantiator used for annotated type.

Above pieces are basically enough to support all three modes of construction @JsonCreator allows (so basically @JsonCreator could be implemented as module, if we wanted!):

  1. "Default" construction that takes no arguments and uses no-argument constructor or factory method
  2. "Delegate-based" construction, in which JSON value is first bound to an intermediate type (such as java.util.Map or Jackson JsonNode), and this instance is passed to single-argument creator method
  3. "Property-based" construction, in which one or more named values (JSON properties) are bound to specified types that match creator arguments, and these are passed to creator method.

Mapping of above construction methods to ValueInstantiator methods is fairly straight-forward:

  1. Simple no-arguments construction (ValueInstantiator.createUsingDefault()): used if the other construction mechanisms are not available: consumes no JSON properties.
  2. Delegate-based construction (ValueInstantiator.createUsingDelegate(Object)): similar to annotating a single-argument constructor or factory method with @JsonCreator, but NOT specifying argument name with @JsonProperty. If specified (i.e. value instantiator indicates it supports this), JSON value for property is first bound into intermediate (delegate) type, and then this value is passed to delegate creator method. Jackson mapper will handle all the details of initial binding, passing delegate object as the argument.
  3. Property-based construction (ValueInstantiator.createFromObjectWith(Object[] args)): similar to using @JsonCreator with arguments that all have @JsonProperty annotation to specify JSON property name to bind.

It is worth noting that order in which availability of different modes is checked is reverse of above: first a check is made to see if property-based method is available; if not, then delegate-based, and finally default construction.

Since this is possibly the most complicated new feature, I will need to defer a full example to another blog post. But let's consider a very simple ValueInstantiator implementation that just supports the default (no-argument) instantiation:

  class SimpleInstantiator extends ValueInstantiator
  {
    @Override public String getValueTypeDesc() { // only needed for error messages
      return MyType.class.getName();
    }

    @Override // yes, this creation method is available
    public boolean canCreateUsingDefault() { return true; }

    @Override
    public MyType createUsingDefault() {
      return new MyType(true);
    }
  }

and similarly you can add support for delegate- or property-based methods.

3. Other notable features

Aside from above-mentioned major features, there are many other useful improvements:

  • "mini-core" jar (jackson-mini-1.9.0.jar)
  • DeserializationConfig.Feature.UNWRAP_ROOT_VALUE
  • @JsonView for JAX-RS methods to return a specific JsonView
  • Terse(r) Visibility: ObjectMapper.setVisibility(), VisibilityChecker.with(Visibility)
  • Add standard naming-strategy implementation(s)
  • Add JsonTypeInfo.defaultSubType property to indicate type to use if class id/name missing
  • Add SimpleFilterProvider.setFailOnUnknownId() to disable throwing exception on missing filter id

"Mini core": as name suggests, there is now a new jar (jackson-mini-1.9.0.jar) that is about 40% smaller than the default one -- about 136kB or so. Size reduction is achieved by leaving out text files (LICENSE), as well as annotations, but otherwise functionality is equivalent to standard core package, i.e. supports streaming API (JsonParser/JsonGenerator, JsonFactory).

DeserializationConfig.Feature.UNWRAP_ROOT_VALUE is counterpart to SerializationConfig.Feature.WRAP_ROOT_VALUE; and there is also now a new annotation -- @JsonRootName -- that can be used to use custom wrapper name instead of the simple class name. This is useful with interoperability, as some frameworks insist on adding such wrappers.

One of few improvements to JAX-RS provider is that now you can add @JsonView annotation to JAX-RS resource methods, and if one is found, it will be set as the active Serialization View during serialization of the result value.

One nice ergonomic improvement is the ability to use much more compact configuration methods for changing default introspection visibility levels.
For example, you can use:

  objectMapper.setVisibility(JsonMethod.FIELD, JsonAutoDetect.Visibility.ANY);

to make all fields auto-detectable, regardless of their visibility. Or, to prevent all auto-detection, you could use:

  objectMapper.setVisibilityChecker(m.getVisibilityChecker()
  	.with(JsonAutoDetect.Visibility.NONE));

An improvement to naming strategy support is inclusion of one "standard" naming strategy -- CAMEL_CASE_TO_LOWER_CASE_WITH_UNDERSCORES -- which converts between standard Java Bean names (that setters and getters use), and C-style names (like used by Twitter). You can enable this converter by:

  mapper.setPropertyNamingStrategy(PropertyNamingStrategy.CAMEL_CASE_TO_LOWER_CASE_WITH_UNDERSCORES);

and from there on, can consume JSON like:

 { "first_name" : "Joe" }

to bind to class like:

public class Name { public String firstName; }

without having to use @JsonProperty to fix name mismatch.

As to sub-typing, you can now use new @JsonTypeInfo property defaultSubType to indicate, as name suggests, default sub-type to use in case where type name was missing or could not be resolved: use it like:

  @JsonSubType(use=Id.NAME, include=As.PROPERTY, defaultSubType=GenericImpl.class)
  public abstract class BaseType { }

And finally, one improvement to Json Filter functionality is ability to specify that it is ok to use a filter id that does not refer to an actual filter (i.e. can not be resolved by the currently configured filter provider) -- use 'SimpleFilterProvider.setFailOnUnknownId(false)' to make this the default behavior. Missing filter is then assumed to mean "no filtering", that is, serialization is handled as if no filter was specified.

Wednesday, September 28, 2011

Advanced filtering with Jackson, Json Filters

I wrote a bit earlier on "filtering properties with Jackson". While it was comprehensive in that all main methods of filtering were covered, there wasn't much depth. Specifically, only very basic usage of Json Filters (@JsonFilter annotation, SimpleFilterProvider as provider) was considered. This approach does allow more dynamic filtering than, say, @JsonView, but it is still somewhat limited. So let's consider more advanced customizability.

1. Refresher on Json Filters

Ok, so the basic idea with Json Filters is that:

  1. Classes can have an associated Filter Id, which defines logical filter to use.
  2. A provider is needed to get the actual filter instance to use, given id: this will be configured by assigning a FilterProvider (such as 'SimpleFilterProvider') to ObjectMapper or ObjectWriter.
  3. Jackson will dynamically (and efficiently) resolve filter given class uses, dynamically, allowing per-call reconfiguration of filtering.

From this it is clear that there are 2 main things you can configure: mechanism that is used to find Filter id of a given class, and mechanism used for mapping this id to actual filter used (implementation of which can be as complicated as you want).

So let's have a look at both parts.

2. Configuring mapping from id to filter instance

Of mechanisms, latter one may be easier to understand and use: one just has to implement 'FilterProvider', which has but one method to implement:

  public abstract class FilterProvider {
    public abstract BeanPropertyFilter findFilter(Object filterId);
  }

given this, 'SimpleFilterProvider' is little more than a Map<String,BeanPropertyFilter>, except for adding couple of convenience factory methods that build 'SimpleBeanPropertyFilter' instances given property names, so you typically just instantiate one with calls like:

  SimpleBeanPropertyFilter filter = SimpleBeanPropertyFilter.filterOutAllExcept("a"));

which would out all properties except for one named "a". This filter is then configured with ObjectMapper like so:

  FilterProvider fp = new SimpleFilterProvider().addFilter("onlyAFilter", filter);
  objectMapper.writer(fp).writeValueAsString(pojo);

which would, then, apply to any Java type configured to use filter with id "onlyAFilter".

3. Configuring discovery of filter id

From above example we know we need to indicate classes that are to use our "onlyAFilter". The default mechanism is to use:

  @JsonFilter("onlyAFilter")
  public class FilteredPOJO {
    //...
  }

But this is just the default. How so? The way Jackson figures out its annotation-based configuration is actually indirect, and fully customizable: all interaction is through configured 'AnnotationIntrospector' object, which amongst other things defines this method:

  public Object findFilterId(AnnotatedClass ac);

which is called when serializer needs to determine id of the filter to apply (if any) for given class. Since the default implementation (org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector) has everything else working fine, what we can do is to sub-class it and override this method.
For example:

  public class MyFilteringIntrospector extends JacksonAnnotationIntrospector
  {
    @Override
    public Object findFilterId(AnnotatedClass ac) {
      // First, let's consider @JsonFilter by calling superclass
      Object id = super.findFilterId(ac);
      // but if not found, use our own heuristic; say, just use class name as filter id, if there's "Filter" in name:
      if (id == null) {
        String name = ac.getName();
        if (name.indexOf("Filter") >= 0) {
          id = name;
        }
      }
      return id;
    }
  }

Above functionality is just to show what is possible, not that it makes sense. Alternatively you could of course define your own annotations to check; or have List of known class names, check class definition or interfaces type implements. The main point is just that you are not limited to using @JsonFilter annotation, but can use pretty much any logic you want, within limits of your coding skills.

The only caveat is that the resolution from Class to matching id is only guaranteed to be called once per ObjectMapper; so any variation in filtering of specific class needs to happen at either mapping of id to filter, or within filter itself.

4. Don't be afraid of sub-classing (Jackson)AnnotationIntrospector

Actually, the key take away might as well be the fact that AnnotationIntrospector is designed to be customizable. It was initially created to allow easy reuse of JAXB annotations (via JAXBAnnotationIntrospector; combining things with AnnotationIntrospector.Pair); but it is also a very powerful general-purpose customization mechanism. But at this point quite underused one at that.

5. Addendum

Some additional notes based on feedback I received:

  • Custom BeanPropertyFilter implementations are obviously powerful too: not only can they completely change what (if anything) gets written for property, they can base this on all configuration accessible via SerializerProvider which is passed to serializeAsField(): for example, it can check to see what serialization view is available by calling 'provider.getSerializationView()'.

Friday, August 12, 2011

Traversing JSON trees with Jackson

1. Three models to rule the...

One of three canonical JSON processing models, tree model, may look a bit like a red-headed stepchild. The amount of effort so far spent on both developing and documenting Jackson data-binding functionality is an order of magnitude higher than all the work for tree model functionality. And considering how much more effort using stream-based processing takes, surprisingly many developers choose it over tree handling.

2. Why I never really liked tree model that much

I confess to having slight aversion to using JSON trees as well; but I have a reasonable excuse: I grow to hate tree-based models with XML. Having survived bad experiences of XML DOM processing (which is both cumbersome and inefficient at same time) tends to inoculate one against further infections. I know this is bit of unjustified bias, considering that most problems with DOM had nothing to do with the basic idea of an in-memory tree model (and not all even due to it being XML...)

3. ... even though I perhaps should have

But Jackson actually does provide reasonable support for JSON trees with its JsonNode based model, and many brave developers have put it to good use. And due to Jackson's extensive support for efficient conversions between models (that is, ability to both combine approaches and to convert data as needed), you don't have to pick and choose just one model but can combine strengths of each model. Tree model's expressive power is actually very useful when doing pre- or post-processing of data binding; or when building quick prototype systems.

4. Basics

A "JSON tree" is defined by one simple thing: org.codehaus.jackson.JsonNode object that acts as the tree of the logical tree. The root node is usually of type 'ObjectNode' (and represents JSON Object), but most operations (all read-operations, specifically) are exposed through basic JsonNode interface.

There are three basic options for creating a JSON tree instance, all accessible via ObjectMapper:

  1. Parse from a JSON source: JsonNode root = mapper.readTree(json);
  2. Convert from a POJO: JsonNode root = mapper.valueToTree(pojo); // special case of 'ObjectMapper.convertValue()'
  3. Construct from scratch: ObjectNode root = mapper.createObjectNode();

The choice largely depends on use case, that is, what do you have to work with; whether you generating new tree from scratch, or modify an existing JSON structure.

After you have the root node you can traverse it modify structure, and convert to other representations (serialize as JSON, convert to a POJO).

5. Back & Forth

Aside from the ability to convert a POJO to a tree, you can easily do the reverse using "ObjectMapper.treeToValue()". Or, if you happen to need a JsonParser, use "ObjectMapper.treeAsTokens()". And to create actual textual JSON, the regular "ObjectMapper.writeValue()" works as expected.

In fact, from ObjectMapper's perspective, JsonNode is just another Java type and is handled using serializers, deserializers which can be overridden if you want to customize handling. You can even replace JsonNodeFactory that ObjectMapper uses, if you want to provide custom JsonNode implementation classes!

6. More convenient traversal

One of things that has quietly improved over time has been traversal. Earliest Jackson versions just supported basic traversal like so:

  JsonNode root = mapper.readTree("{\"address\":{\"zip\":98040, \"city\":\"Mercer Island\"}}");
  JsonNode address = root.get("address");
  if (address != null && address.has("zip")) {
    int zip = address.get("zip").getIntValue();
  }

but it soon became apparent that null checks are a worthless hassle, so alternative access, "path()" was quickly added. It allows for traversing over virtual paths, without worrying whether a node exists: if one does not exist, it will just be evaluate as "missing node" when trying to access actual leaf value:

  JsonNode root = ...;
  int zip = root.path("address").path("zip").getValueAsInt(); // if no such path, returns 0
  // could also do:
  JsonNode zipNode = root.path("address").path("zip");
  if (zipNode.isMissingNode()) { // true if no such path exists
  }

This is fine and dandy for read-only use cases, but it does not help when trying to add things -- while you can traverse path that does not really exist, you can not add anything to it. To address this shortcoming, Jackson 1.8 comes equipped with "with()" method, which will actually create the path if it does not exist. So you can finally write something like this:

  JsonNode root = ObjectMapper.createObjectNode();
  // note: JsonNode.with() returns 'JsonNode'; but ObjectNode.with() 'ObjectNode' -- go contra-variance!
  root.with("address").put("zip", 98040);

which actually makes Jackson Tree usage almost as convenient as I would like it to be. It is especially useful when materializing full trees from scratch: you can implicitly build the tree structure just by traversing it!

7. More?

Jackson tree model is still somewhat spartan, especially compared to features galore of data binding. Going forward it would be nice to add support for things like:

  • Simple path language (JsonPath, JsonQuery?) support, to be able to evaluate expressions to locate nodes.
  • Filtering during construction, to create trimmed/pruned trees, sub-trees
  • More advanced find methods? There already exists a few "findXxx()" methods in JsonNode, but more would make sense, esp. with configurable matchers or filters
  • Method names are bit too verbose (mostly due to historical reasons -- I didn't realize early enough that long method names can hurt when chained calls are used)

But as usual, much of Jackson development work is feedback-driven -- features that get used also get more likely further improved. So if you do find Tree Model useful, let development team know that!

Thursday, August 11, 2011

One of coolest, least well-known Jackson features: Mr Bean, aka "abstract type materialization"

1. Quest for simplest JSON processing, eliminating monkey code: "struct classes"

I have found myself using "Java structs" quite often, when accessing JSON services from Java. By this I mean simple public-field-only classes like:


public class RequestDTO {
 public long requestId;
 public String callerId;
}

While many Java newbies think there is something wrong in using public fields, there is actually very little harm in using such classes for simple data transfer, if no actual business logic is needed for classes themselves.

2. But sometimes "real" classes would be nice

Then again, sometimes it would be nice to use more full-featured Bean(-like) POJOs. Perhaps we want to add some input validation for setters; or add convenience accessors, or even just occasional 'toString()' implementation.

For above example, we might want to get something like:


public class RequestImpl
{ private long requestId; private String callerId; public RequstImpl() { } public long getRequestId() { return requestId; } public String getCallerId() { return callerId; } public void setRequestId(long l) { requestId = l; public void setCallerId(String s) { callerId = s; } @Override public String toString() { return String.format("[request: id %d, caller %s]", requestId, callerId); } }

But ideally we would usually just define something like


public interface Request {
  public long getRequestId();
  public String getCallerId();

  public void setRequestId(long l);
  public void setCallerId(String s);
}

and somehow get an implementation; alas, that usually means writing boiler-plate implementation for that interface (and if we are masochists, sometimes even intermediate abstract classes...)

So what's the problem here? I don't particular like writing monkey code to declare basic setters, getters, and fields; especially when there is nothing interesting going on there, just mechanical typing. And while one can use IDEs to generate sources, this only helps with bootstrapping: you still get more source code to maintain, which translates to more place where bugs may hide when definitions are edited. Similarly various annotation-based post-processors seem alien to me if they just produce more source code to compile.

3. So why not just like... get implementations "materialize"?

But while I don't like the idea of getting yet more source code generated to be compiled, maintained, I do like the idea of getting actual implementation classes dynamically.

And this is where entry #6 of "7 Jackson killer features" comes in: enter mr. Bean! When enabled, it can actually materialize concrete implementations as needed.

4. Mr Bean: basics

(from FasterXML Mr Bean Wiki page)

Basic usage is simple: you need jackson mrbean jar (included in Jackson distribution), and need to enable functionality with:


  ObjectMapper mapper = new ObjectMapper();
  mapper.registerModule(new MrBeanModule());

and then just watch interfaces appear: for example, with above example:


  Request request = objectMapper.readValue(jsonInput, Request.class); // where Request is an interface

What happens here is that mr Bean extension hooks with ObjectMapper, and whenever an abstract type is encountered and there is no concrete class available (no abstract type mapping; no annotation to indicate concrete type; no @JsonTypeInfo to provide subtype information), it is asked to "materialize" concrete type.

Materialization simply means generating bytecode using ASM, based on getters and/or setters; adding necessary internal fields, loading class and returning it to caller. After this, core Jackson mapper can introspect all information it needs, and what you get is an instance of this implementation. Implementations are cached for later use, and performance-wise they behave similarly to manually implemented ones would.

5. Mr Bean: but wait! There's more!

Ok: so we can get monkey code materialized: getters and setters are implemented, and internal fields added to store values. But this is just the beginning.

First: if you do not need to use setters yourself you can freely omit them from interface definition.
Mr Bean is smart enough to figure out that setters are typically needed to set values (or public fields) if there are getters materialized.
So you can simplify your interfaces/abstract classes to look something like:


  public interface RequestWithoutSetters {
    public long getRequestId();
    public String getCallerId();
  }

and things will still work just fine; you can't access setters (which actually may be a good thing), but Jackson data binder can populate values just fine (internally either setters get generated; or public fields added to implementation, this is an implementation detail).

Aside from simplistic get/set Bean it is more commont to want a partial implementation; an abstract class where you provide some methods and/or fields, but can leave implementation of trivial properties to Mr Bean. This it can do just fine: mr Bean can materialize abstract classes, just "filling in the blanks".

So you can ask for a class like:


  public abstract class RequestBase {
    public long getRequestId();
    public String getCallerId();
    
    @Override public String toString() {
      return String.format("[request: id %d, caller %s]", requestId, callerId); }
    }
  }

and things work, well, as expected. Note, too, that you can implement setters and getters, not just "other" methods.

And finally: you can use annotations normally as well, adding them to your interface/abstract class definition. Thanks to Jackson's powerful and versatile annotation handling (including annotation "inheritance" for methods), you can do something like:


  // JSON we get has weird names; need to annotate
  public abstract class RequestBase {
    @JsonProperty("REQID")
    public long getRequestId();
    @JsonProperty("CALLERID")
    public String getCallerId();
  }

and get things configured as per annotations.

6. Known issues?

Mr Bean seems to work to degree I need it to work. But there are some potential concerns you may need to be aware of:

  • Jackson has multiple ways of dealing with abstract types: do you want bean materialized or not? As mentioned above, mr Bean does not try to materialize abstract types that seem to expect different kind of handling; for example, if interface has @JsonTypeInfo annotation, assumption is that polymorphic handling can figure out actual type. But it is possible that there are corner cases (esp. when using "default typing") there might be conflicts. So polymorphic types may not mix well with mr Bean materialization
  • Generic signatures may not be added as expected. Although you can declared generic types for abstract methods just fine, and Jackson mapper should fine declarations, there are some issues due to complexities in getting generic declarations work with ASM. You may need to use additional annotations (@JsonDeserialize(contentAs=...)) in some cases.

Above is just a list of potential concerns -- as far as I know, they haven't been found to be much of a problem in actual use so far.

7. What Next?

Usage, usage, usage! It would be great to get more Jackson users use this potentially hugely work-saving feature. And if you find the feature useful, make sure to let your friends know! (if you hate it, just let me know :-) ).

Tuesday, July 26, 2011

Jackson tips: using @JsonAnyGetter/@JsonAnySetter to create "dyna beans"

One relatively common "special" POJO is so-called dynamic bean ("dyna-bean"), which is sort of combination of regular bean and basic Java Map; with zero or more properties with known name, and extensible set of 'other' key/value pairs.

Here's what such a POJO might look like:


public class DynaBean
{
    // Two mandatory properties
    protected final int id;
    protected final String name;

    // and then "other" stuff:
    protected Map<String,Object> other = new HashMap<String,Object>();

    public DynaBean(int id, String name)
    {
        this.id = id;
        this.name = name;
    }

    public int getId() { return id; }
    public String getName() { return name; }

    public Object get(String name) {
        return other.get(name);
    }

    public void set(String name, Object value) {
        other.put(name, value);
    }
}

Since Jackson can serialize Bean as well as Maps, what is the problem? As presented, bean would not serialize and deserialize as expected, although it could be modified to just return Map of "other" properties, and deserialize them back. This would work, but would result in an additional level of wrapping, so that secondary properties would be within a separate JSON Object.

But Jackson can actually be made to work with such POJOs: here is one way to do it:


public class DynaBean
{
    // Two mandatory properties
    protected final int id;
    protected final String name;

    // and then "other" stuff:
    protected Map<String,Object> other = new HashMap<String,Object>();

    // Could alternatively add setters, but since these are mandatory
    @JsonCreator
    public DynaBean(@JsonProperty("id") int id, @JsonProperty("name") String name)
    {
        this.id = id;
        this.name = name;
    }

    public int getId() { return id; }
    public String getName() { return name; }

    public Object get(String name) {
        return other.get(name);
    }

    // "any getter" needed for serialization    
    @JsonAnyGetter
    public Map<String,Object> any() {
        return other;
    }

    @JsonAnySetter
    public void set(String name, Object value) {
        other.put(name, value);
    }
}

And there we have it: serializes and deserializes nicely.

Share and enjoy...

Monday, July 04, 2011

Jackson Annotations: @JsonCreator demystified

One of more powerful features of Jackson is its ability to use arbitrary constructors for creating POJO instances, by indicating constructor to use with @JsonCreator annotation. A simple explanation of this feature can be found from FasterXML wiki; but it only scratches surface of all the power this annotation exposes. This article will expand on what can be done with this annotation.

1. What are Creators in Jackson?

"Creator" refers to two kinds of things: constructors, and static factory methods: both can be used to construct new instances. Term "creator" is used for convenience, to avoid repeating "constructor or static factory method".

In some places term "creator method" may be used, although this is not preferred (since constructors are not regular methods). In case of static factory methods, method's return type must be same as declaring classes type, or its subtype.

2. Two kinds of creators: property-based, delegate-based

There are two kinds of creators that one can denote using @JsonCreator: property- and delegate-based creators:

  • Property-based creators take one or more arguments; all of which MUST be annotated with @JsonProperty, to specify JSON name used for property. They can only be used to bind data from JSON Objects; and each parameter represents one property of the JSON Object; type of property being used for binding data to be passed as that parameter when calling creator.
  • Delegate-based creators take just one argument, which is NOT annotated with @JsonProperty. Type of that property is used by Jackson to bind the whole JSON value (JSON Object, array or scalar value), to be passed as value of that one argument.

In addition, sometimes distinction is made between two kinds of delegate-based creators: those that take scalar value (int/Integer/long/Long, string or boolean/Boolean) and other delegates. There is only one important distinction from user perspective: there is no creator overloading, so only one creator of each type is alllowed. More on this later on.

3. Property-based creators

Property-based creators are typically used to pass one or more obligatory parameters into constructor (either directly or via factory method). If a property is not found from JSON, null is passed instead (or, in case of primitives, so-called default value; 0 for ints and so on).

A typical use case could look like this:


  public class NonDefaultBean {
    private final String name;
    private final int age;

    private String type;
  
    @JsonCreator
    public NonDefaultBean(@JsonProperty("name") String name, @JsonProperty("age") int age)
    {
      this.name = name;
      this.age = age;
    }

    public void setType(String type) {
      this.type = type;
    }
  }

where two properties are passed via constructor creator; and then a third one through regular setter. Note that we could have as well used a static factory method as creator.

One thing to note about these creators is that it is possible to create a sub-type of class: this allows implementing polymorphic handling manually if necessary, although with limitation that creator itself must be in base class. But all other properties can be passed to sub-classes, as Jackson is smart enough to check type and properties of the actual instance, not just declared nominal type.

4. Delegate-based creators

Whereas you can use multiple arguments with properties-based creators (but must explicitly name them), delegate-based creator takes just one argument. Jackson will then use type of that argument for data-binding, and bind JSON to that type before calling creator. Creator is then free to construct instance any way it wants to. A typical usage looks like:


  public class ObjectDelegateBean
  {
    private final String name;
    private final int age;
    private final String type;

    @JsonCreator
    public ObjectDelegateBean(Map<String,Object> props)
    {
      name = (String) props.get("name");
      age = (Integer) props.get("age");
      type = (String) props.get("type");
    }
  }  

In this case we just manually extract properties from a Map (where JSON data has been bound to "natural" types; Maps, Lists, String, Numbers and Booleans). Other common delegate types used are JsonNode (to use JSON tree as intermediate form), TokenBuffer (stream of exact JsonTokens, which can be used to create a JsonParser) and basic java.lang.Object (which would map to natural types mentioned earlier).

But as I mentioned earlier, it is also possible to use variants to handle scalar types: specifically ints (or Integers), long (or Longs) and Strings (in future support will also be added for double/Double).

For example:


  public class DateBean
  {
    private Date date;

    private DateBean(Date date) {
      this.date = date;
    }

    @JsonCreator
    public static DateBean factory(long timestamp) {
      Date d = new Date(timestamp);
      return new DateBean(d);
    }
}

which is bit contrived example, as Jackson can easily bind java.util.Date directly. However, some JSON formats support overloading, or try to minimize size; and JSON Strings are sometimes used to bundle semi-structured data

One point to note about "scalar delegates" is that Jackson allows overload of creators, as long as there is only one creator type per JSON type: so you can have only one properties-based creator (or delegate-based creator that takes something other than match of JSON scalar type) , but there can be an additional string-delegate creator as well as an int-delegate creator. Limitation is conceptually simple: there must only be one applicable delegate to choose from, knowing type of JSON value being bound -- and this is why delegates that take String, Integer/int or Long/long are special cases of delegate-based creators.

5. More to know?

In near future (Jackson 1.9) there will be significant improvements in this area: new things called "value instantiators" will be allowed to handle feature set similar to that of @JsonCreator; but that need not be bound to existing Java constructors or factory methods.

Friday, July 01, 2011

Oh yes, Jackson 1.8 was released a while ago... :-)

Whoa. Looks like I forgot to blog about something rather big; the release of Jackson 1.8.0. This is mostly because I did that shortly before going for a nice long vacation, and by the time I came back, I had forgotten most of it. Although was reminded by the release in form of a nice long list of new bug reports. :-)

At any rate, I did write something about the release; so as a background, here are contemporary accounts of the event:

Actually, both of above enumerate all the new features, so I can't add much more at detailed level.

But one thing that may not be obvious is that 1.8 was a huge step forward to allow full power of configurability for Jackson Modules; concept which was added in 1.7, but that really became useful enough for major extensions in 1.8. As a result all the exciting modules (esp. ones at FasterXML GitHub) -- including "Jackson XML module" for XML-based serialization, deserialization; Hibernate Jackson module, and even Jackson Scala module -- require 1.8 as their baseline. And at least XML and Scala modules were driving some of improvements made to module interface.

Another main goal was to focus on paying down sort of "feature debt" -- something similar to technical debt, but relating to the fact that sometimes oldest feature-requests tend to be forgotten, and development focuses more on latest ideas. In 1.8, then, we saw completion of many long-standing (and sometimes, long-ignored) feature requests.

Anyway -- it has been a while since 1.8.0 was released; and at this point I am focused on starting work on 1.9.0. While only one new feature has been completed, there is lots of work behind the scenes; much of which is aimed at helping modules (or working on specific modules). But more on these ventures in future blog posts; stay tuned!

Friday, March 11, 2011

Jackson 1.8: custom property naming strategies

One of big goals for Jackson 1.8 is to implement oldest open feature requests. One such feature is ability to customize property naming strategy: that is, to allow use of JSON names that do not match requirements of Bean naming conventions.

For example, consider case of Twitter JSON API, which uses "C-style" naming convention, so what would be "profileImageUrl" with bean naming convention, is instead "profile_image_url".
With earlier Jackson versions, one has had to annotate all properties with @JsonProperty annotation; or use rather unwieldy method names like "getprofile_image_url()" (which would work, just look ugly).

But version 1.8 will finally allow use of custom naming strategies. Let's examine how.

First thing to do is to extend org.codehaus.jackson.map.PropertyNamingStrategy

  static class CStyleStrategy extends PropertyNamingStrategy
  {
    public String nameForField(MapperConfig?> config, AnnotatedField field, String defaultName) {
return convert(defaultName); } public String nameForGetterMethod(MapperConfig?> config, AnnotatedMethod method, String defaultName) { return convert(defaultName); } public String nameForSetterMethod(MapperConfig?> config, AnnotatedMethod method, String defaultName) { return convert(defaultName); } private String convert(String input) { // easy: replace capital letters with underscore, lower-cases equivalent StringBuilder result = new StringBuilder(); for (int i = 0, len = input.length(); i < len; ++i) { char c = input.charAt(i); if (Character.isUpperCase(c)) { result.append('_'); c = Character.toLowerCase(c); } result.append(c); } return result.toString(); } }

which in this case will just convert property names by replacing all capital letters with underscore followed by lower-cased version of the letter.
And then the only other thing is to register this strategy with ObjectMapper:

   ObjectMapper mapper = new ObjectMapper();
   mapper.setPropertyNamingStrategy(new PrefixStrategy());

and after this we could do:

  static class PersonBean {
        public String firstName;
        public String lastName;

        public PersonBean(String f, String l) {
            firstName = f;
            lastName = l;
        }
  }

  // so this would hold true:
  assertEquals("{\"first_name\":\"Joe\",\"last_name\":\"Sixpack\"}", new PersonBean("Joe", "Sixpack"));

This should help working with APIs that use C-style underscore notation, or variation of camel casing (such as one where first word is also capitalized).

Thursday, March 10, 2011

Upgrade from org.json to Jackson, piece by piece, using jackson-module-org-json

1. Background

One common task for software engineers is upgrading legacy systems; things written a while ago, often by developers who have since moved on, using tools and techniques thought to be good fit back then. While the reason for upgrades are typically either addition of minor new features, or fixes to the most annoying issues, it can and should also include some pro-active work to improve quality of the system itself, to "pay off technical debt" before it is due.

One of first things you usually observe, when looking under the hood of such systems is that various design and implementation choices do not look so good any more. Sometimes technology choice turned to be a dead-end; sometimes design was based on incomplete or incorrect understand. But almost always some of supporting libraries have become obsolete: tool that was thought as best-of-breed back then may well now be amongst most despised pieces of software ever deviced.

2. Problem

One common case of "library rot" that I have often encountered is that of using "org.json" library (from http://json.org/java) for reading and writing JSON. It was the first Java JSON library around, and served as sort of proof of concept back in the early days of JSON usage. But time has not been kind to it: with arrival of more modern, fully-featured alternatives, it is the monoplane of the second world-war era; once serviceable, but now just cumbersome and slow thing that few would choose to use given a choice of modern JSON libraries like Jackson. You can get improvements in about any area, from convenience and intuitiveness to performance, just by upgrading.

When considering possible upgrate of the JSON library, the initial issues encountered are usually not very big: changing code that directly deals with reading and writing JSON is not awfully hard, and typically resulting code is much cleaner. But there is often much bigger problem outside this core JSON code: in case of org.json package, specifically, all processing is done using concrete types (JSONObject, JSONArray). And values of these types are very easy to leak outside, resulting in unintended tight coupling of non-JSON-processing code with the JSON parser implementation. While this may be a good lesson on values of proper design and encapsulation of concerns, it is of little consolation when you are looking at code that had neither.

If your only choice is to rewrite major portions of the legacy system (to mostly address relatively small JSON-specific portion), it is often easier to just let things be as they are.

3. But what if we only upgraded parser, not JSONObject/JSONArray?

Thinking about this a bit, I realized that the biggest immediate obstacle is really deep coupling via 2 value types. So what if we just continued to use these non-abstract "abstractions", but replace underlying JSON parser and generator? While this would not allow use of more convenient and proper real abstractions (of data binding to POJOs), at least it could give some nice efficiency improvements (more on this below); and make it easier to do complete conversion bit later on. Killing a beast is often easiest by thousand cuts.

Fortunately for us, Jackson 1.7 made it very easy to add support for additional data types; so I decided to spend one (and only one!) evening to write an extension module -- jackson-module-json-org available from GitHub as https://github.com/FasterXML/jackson-module-json-org -- to allow using Jackson to effectively read and write JSONObject and JSONArray types.

Usage is rather straight-forward; after registering new module with ObjectMapper

  ObjectMapper mapper = new ObjectMapper();
  mapper.registerModule(new JsonOrgModule()); // package 'com.fasterxml.jackson.datatype.jsonorg'

you can use all the usual conversions like so:

  // read/write JSON
  JSONObject ob = mapper.readValue(json, JSONObject.class); // read from a source
  String json = mapper.writeValue(ob); // output as String

  // convert POJOs to/from JSONObject/JSONArray
  MyValue value = mapper.convertValue(jsonObject, MyValue.class);
  JSONObject jsonObject = mapper.convertValue(value, JSONObject.class);

  // and even conver to/from Jackson Tree model:
  JsonNode root = mapper.valueToTree(jsonObject);
  jsonObject = mapper.treeToValue(root, JSONObject.class);

and there you have it, convenient conversions between JSON, POJO, and org.json types.

In fact, this will also help with later stages, because you can easily (and relatively efficiently) go back and forth between representations: so we can extend "org.json-free" zone further and further away from the source. This is what I really mean by piece-by-piece approach: it is possible to do refactoring one component or area at a time.

4. But wait! There is more.... performance bonus!

By now it should be well-known that Jackson is the most performant library for doing JSON manipulation in Java. So could it be that using this module might speed up processing as well? Aren't we still using some org.json pieces? Turns out that since JSONObject is little more than a wrapper around basic Java HashMap (and JSONArray similarly around ArrayList), most performance benefits from different parser and generator. So I went ahead and modified my JSON parser performance test to compare "raw Jackson" (using Jackson Streaming API), "Jackson with Tree Model" (JsonNode), "Jackson with org.json" and stock org.json. Here is one sampling of results I saw on my machine (using sample document #5 from org.json):


Test 'Jackson, stream' -> 282 msecs
Test 'Jackson, JsonNode' -> 411 msecs
Test 'Jackson + module-org-json' -> 393 msecs
Test 'Json.org' -> 783 msecs

It turns out that while Streaming API is the fastest way, the new "jackson-module-json-org" extension can actually bind JSON as fast as Jackson Tree model -- both of which are twice as fast as basic org.json package for this small document (for bigger documents difference is typically even bigger).

So in addition to cleaning up the code, you may also end up speeding up your JSON processing code by 100%. Not a bad deal, eh?

Saturday, February 05, 2011

Every day Jackson usage, part 3: Filtering properties

(part of continuing series on covering common usage patterns with Jackson JSON processor; previous entry covered handling open (extensible) content)

One of first things users eventually want to configure when using Jackson is (re)defining which Java object properties to serialize (written out) and which not.

1. What gets serialized by default?

Properties of an object are initially determined by process called auto-detection: all member methods and fields are checked to find:

  1. "Getter" methods: all no-argument public member methods which return a value, and conform to naming convention of "getXxx" (or "isXxx", iff return type is boolean; called "is-getter") are considered to infer existence of property with name "xxx" (where property name is inferred using bean convention, i.e. the leading capitali letter(s) is changed to lower case
  2. field properties: all public member fields are considered to represent properties, using field name as is.

In case both a getter and a field is found for same logical property, getter method has precedence and is used (field is ignored).

Set of properties introspected using this process are considered to be the base set of properties. But the auto-detection process itself can be configured differently, and there are multiple annotations and configuration settings that can further change actual effective set of properties to serialize.

2. Changing auto-detection defaults: @JsonAutoDetect

If the default auto-detection visibility limits (fields and member methods needing to be public) are not to your liking, it is easy to change them by using one of following method:

  • @JsonAutoDetect annotation can be defined for classes; properties "fieldVisibility", "getterVisibility" and "isGetterVisibility" define minimum visibility needed to include property (for fields, getters and is-getters, respectively). It is possible to, for example, include all field properties regardess of visibility (@JsonAutoDetect(fieldVisibility=Visibility.ANY)); or to disable getter-method auto-detection (@JsonAutoDetect(getterVisibility=Visibility.NONE)), and combinations there-of. Note that this annotation (just like any other Jackson annotation) can be applied as a mix-in annotation, without having to modify type directly; and can be added to a base type to apply to all subtypes.
  • ObjectMapper.setVisibilityChecker() can be used to define customized minimum visibility detection

Changing minimum auto-detection visibility limits is an easy to way to increase number of properties discovered (for example by exposing all member fields; similar to how libraries like XStream and Gson work by default), or to prevent any and all auto-detection (i.e. to force explicit annotations using @JsonProperty or @JsonGetter annotation).

As an example, to serialize all fields (and use no getter methods), you could do:

  @JsonAutoDetect(fieldVisibility=Visibility.ANY,
                  getterVisibility=Visibility.NONE, isGetterVisibility=Visibility.NONE)
  public class FieldsOnlyBean {
    private String name; // will now be used instead of getName()
    public String getName() { throw new Error(); } // never used!
  }

3. Explicitly ignoring properties: @JsonIgnore, @JsonIgnoreProperties

Given set of auto-detect potential properties is the starting point; and it can further be modified by per-property annotations:

  • @JsonProperty (and @JsonGetter, @JsonAnyGetter) can be used to indicate that a field or method is to be consider a property field or getter method, even if it isn't auto-detected.
  • @JsonIgnore can be used to forcibly prevent inclusion, regardless of auto-detection (or other annotations)

In addition, there is per-class annotation @JsonIgnoreProperties that can be used alternatively to list names of logical properties NOT to include for serialization; it may be easier to use via mix-in annotations than per-property annotations (although both can be used via mix-in annotations).

So you could do:

  @JsonIgnoreProperties({ "internal" })
  public class Bean {
    public Settings getInternal() { ... } // ignored
    @JsonIgnore public Settinger getBogus(); // likewise ignored
    public String getName(); // but this would be serialized
}

4. Defining profiles for dynamic ignoral: JSON Views (@JsonView)

So far configuration methods have been applied statically; meaning that a property will either always be included (except for special case of possibly suppressing null-values), or never included.

JSON views are a way to define more dynamic inclusion/exclusion strategy. The idea is to define inclusion rules for properties, by associating logical views (classes used as identifiers; this allows use of hierarchic views) with properties, using @JsonView annotations; and then specifying which view is to be used for serialization. This is often used to define smaller "public" set of properties, and larger "private" or "confidential" set of properties. See @JsonView wiki page for usage example.

5. Ignoring all properties with specified type: @JsonIgnoreType

In addition to defining rules on per-property basis, there are times when it makes sense to just prevent serialization of any auto-detected properties for given type(s). For example, many frameworks add specific accessors for types they generate, which return objects that should not be serialized.

For example, let's say that an Object-Relational Mapper always adds "public Schema getSchema()" accessors for all value classes. And if this is metadata that is not part of serializable state, we can prevent its inclusion in serialization by adding @JsonIgnoreType annotation on Schema type (or its supertype). This is often easiest done using mix-in annotations.

6. Fully dynamic filtering: @JsonFilter

Although JSON views allow somewhat dynamic filtering, definitions of filters are still static. This means that it is only possible to dynamically choose from a static set of views.

JSON Filters are a way to implement fully dynamic filtering. The way this is done is by defining logical filter a property uses, with @JsonFilter("id") annotation, but specifying actual filter (and its configuration) using ObjectWriter. Filters themselves are obtained via FilterProviders which can be fully custom, or based on simple implementations. Check out JSON Filter wiki page for details.

7 . Most extreme way to filter out properties: BeanSerializerModifier

And if ability to define custom filters is not enough, the ultimate in configurability is ability to modify configuration and configuration of BeanSerializer instances. This makes it possible to do all kinds of modifications (changing order in which properties are serialized; adding, removing or renaming properties; replacing serializer altogether with a custom instance and so on): you can completely re-wire or replace serialization of regular POJO ("bean") types.

This is achieved by adding a BeanSerializerModifier: the simplest way to do this is by using Module interface. Details of using BeanSerializerModifier are more advanced topic; I hope to cover it separately in future. The basic idea is that BeanSerializerModifier instance defines callbacks that Jackson BeanSerializerFactory calls during construction of a serializer.

Related Blogs

(by Author (topics))

Powered By

Powered by Thingamablog,
Blogger Templates and Discus comments.

About me

  • I am known as Cowtowncoder
  • Contact me at@yahoo.com
Check my profile to learn more.