Dashboard > iBATIS DataMapper > Home > iBATIS 3.0 Whiteboard
iBATIS 3.0 Whiteboard
Added by Clinton Begin, last edited by sdimi on Apr 12, 2008  (view change)
Labels: 
(None)


Opportunity for Change.

January 11th, 2008 marks the 4th Anniversary of the iBATIS 2.0. It has served the community well for four years, but times change. The last few years have been full of innovation and shifts in technology and mindset. The impact of frameworks like Ruby on Rails cannot be ignored. The industry has noticed and finally invested in lightweight frameworks, agile principles and simple solutions first.

Luckily, this shift aligns closer toward iBATIS, not further from it.

iBATIS has always been about simplicity. Since the very beginning, with each release of iBATIS we've made the framework smaller and simpler with fewer dependencies. Yet, we've been able to make the framework faster and more powerful every time.

We hope to continue on this promise with iBATIS 3.0. This whiteboard is where it all begins!

How to work with this page.

  • Committers can change the content of this page.
  • Where there is disagreement among committers, all alternatives should be presented until resolved.
  • Anyone in the community can comment on the content using the comment section below.
  • Committers may make changes to the content based on the comments.
  • Comments acted upon may be deleted to keep the comments relevant to the current state.
  • Comments may be archived if they are deemed irrelevant or are declined (avoid deletion).
  • The committers are charged with ensuring the continuity, consistency and focus of the design is managed.
  • Not all comments or suggestions will be accepted.
  • We'll run a tight ship to ensure this page doesn't turn into a zoo.
  • These rules can change.

Themes for iBATIS 3.0

  • Test driven development
  • Code cleanliness over performance
  • Simple design over flexible design
  • One JAR file
  • No required 3rd party dependencies
  • Better plug-in support
  • Third party plug-ins are hosted elsewhere (sub-project on SF or CodePlex) http://sourceforge.net/projects/ibatiscontrib/

The Methodology

  • Clinton says: When I wrote iBATIS 1.0, it was just me. I hacked it out in the most simple and straightforward way possible, with only high level functional tests (actually, they were JPetStore tests). With iBATIS 2.0 I promised myself I'd write unit tests, but I didn't test drive it. The test coverage for 2.0 is around 63%, but looking deeper it's actually not even that good in my own opinion. Therefore, for 3.0 I'd suggest test driving it. The wise man (you) learns from the mistakes of others (me).

Our team is already very good at developing and maintaining the iBATIS codebase. The source code in our repository is always buildable and even deployable. We don't release when it's stable, because it's always stable. We release when we decide we've hit a milestone that together make a compelling feature set for deployment. We're already quite agile.

To ensure that we have clear direction for the development of iBATIS 3.0, here is a summary of our lifecycle.

Practice Agile Concept
Discussion of new feature/issue on the dev and user mailing lists. customer involvement
Creation of JIRA ticket for new feature/issue. user story
Test drive development of new feature/issue. test always
Commit feature/issue with unit test as a single revision. working software
Automated build and test run upon each checkin. continuous integration
Nightly (or more frequent) builds available for download. release often
Release upon milestone defined by useful feature set decided by the team/community. release planning

Design...Feature by Feature

The following sections describe the design of the iBATIS 3.0 features.

Interface Binding

With the introduction of Java 5 we have new opportunities to improve the user experience. The most important being the elimination of ambiguous type casting and string based calls to SQL maps. For example, the current way to call a SQL Map is this:

Employee employee = (Employee)sqlMapper.queryForList("getEmployee", 5);
//...and...
List employees = sqlMapper.queryForList("listAllEmployees");

The problems are obvious. First, in both cases the mapped statement names are string based. This is obviously error prone and ugly. There's no hope for help or hints from the IDE, or even compiler warnings or errors.

The second problem is the type safety of the parameters and results. For example, in the first case, who's to say that the parameter is an integer? And who's to say that you're actually going to get an Employee class back. Again, no compiler warnings or errors. Only runtime exceptions will be reported.

And finally, in the case of collections, the problems are the same. You may get a list of Dogs back, instead of Employees, without warning before runtime.

Overall, it's ugly – still better than JDBC – but not as good as it could be.

With Java 5 we can leverage generics to type the collection and therefore for the first time we can create a consistent API for collections without additional configuration. Instead of declaring types in XML and calling statements via loose string names, we can make effective use of a very descriptive and typesafe Interface type. Nothing new to learn, just a normal interface.

public interface EmployeeMapper {
  Employee getEmployee (int employeeId);
  List<Employee> listAllEmployees();
}

That's it! iBATIS takes care of the implementation of the interface for us.

We can now call mapped statements in a much cleaner and safer way.

Employee emp = empMapper.getEmployee(5);
//...and...
List<Employee> employees = empMapper.listAllEmployees();

No casting, no strings, typesafe parameters and results. The extra code incurred on the interface will be made up by far less configuration in the XML or annotations. At most, we'll have the SQL and any result or parameter mappings that need to be overridden.

The interface tells nearly everything we need to know.

  • Name of the "statement" (same as the method name).
  • Parameter types (if any).
  • Result types (including collection element types).

And most interesting (and possibly controversial) of all...

  • We could even generate the SQL based on conventions in the method name. More about that in the Multilevel Configuration section below.

The bound interface approach also makes it easier for frameworks like Spring to inject just a nice clean mapper class, without exposing the full details of the transaction manager etc.

Transactions, Sessions, and Factories – MapperFactory

So where do we get a "Mapper" instance from? We'll still need some sort of central class for iBATIS configuration and control. The name is up for debate, but for now we'll call it MapperFactory.

In general, the MapperFactory is responsible for transactions and mapper instances. The MapperFactory itself will come from some sort of configuration class (again, more about that in the configuration section below).

MapperFactory factory = someConfiguration.buildMapperFactory();
EmployeeMapper employeeMapper = factory.getMapper (EmployeeMapper.class);

I'm not sure that it gets any easier...but that's it. As for transactions...

  • Clinton says: The concept of a session should be merged with that of a transaction. In iBATIS, session and transaction have largely been one and the same. But they were separated for some artificial architectural reason. There are a few things sessions could offer, but for 80% of cases, they don't matter. Similarly batches were separated from transactions, which caused messy nested try/finally blocks. So lets avoid them for now, unless we find a real need for them.

Transactions will be handled either by iBATIS or some 3rd party framework or container. To date they've been handled by a ThreadLocal instance inside a SqlMapClient instance. We get a lot of questions about thread safety due to this approach, so for no technical reason but perhaps a code clarity reason, we could use the following approach:

Transaction tx = factory.startTransaction();
try {
  // do work
  tx.commit();
} finally {
  tx.end();
}

Batches will be handled by simply starting a batched transaction:

Transaction tx = factory.startBatch();
try {
  // do work
  // balk executes current batch immediately, returns total rows updated 
  // (balks on non-batched transactions)
  int totalRowCount = tx.flush(); 
  // commit executes batch too, returns total rows updated and commits
  totalRowCount = tx.commit(); 
} finally {
  tx.end();
}

Similar to the existing transaction API, we'll still support isolation levels and other transaction options.

Transaction tx = factory.startTransaction(isolationLevel);
try {
  // do work
  tx.commit(); 
} finally {
  tx.end();
}

If a third party is handling transactions, the managed connection can be explicitly passed:

// Can be handled by encapsulating framework (e.g. Spring)
Transaction tx = factory.useConnection(conn); 
try {
  //...do work...
  tx.flush(); // executes batched updates immediately
  // commit simply calls commit on the connection 
  // if local commits are allowed, otherwise it is ignored.
  tx.commit();
} finally {
  tx.end(); // Can also be handled by encapsulating framework (e.g. Spring)
}

Overall this API is cleaner, simpler and is easier to integrate with 3rd party frameworks like Spring. The MapperFactory can be injected into SpringDAOs if access to the transaction manager, batch or isolation levels is needed. To isolate this further, we could separate the transaction manager from the MapperFactory, or even allow the developer to identify their own TransactionManager interface that uses conventions to loosely bind the transaction management methods to their own class (better isolation of the framework).

Multilevel Configuration

By far the most "old-school" aspect of iBATIS of iBATIS is the configuration. When originally built, XML was thought to be a best practice for configuration. Indeed, XML is still a good choice for iBATIS, as it is a great way to code complex multiline SQL. Therefore XML will remain. However, it will not be the default, or even the solution for most cases.

iBATIS 3.0 will use a multilevel configuration approach. That is, there will be a number of ways to configure iBATIS that work together and override each other. The four ways, in order of application, are:

  • Convention
  • Annotation (overrides convention)
  • XML (overrides convention and annotation)
  • Java API (overrides convention, annotation and XML)

Configuration by Convention

  • Clinton says: This is the biggest departure from our former principles. Anyone who knows me knows I'm not a big fan of generation. But if we're to adapt to changing best practices, we must consider simple solutions for even complex applications. Rest assured, that people who don't like convention based configuration will be able to easily ignore it. I believe it will be very useful, especially for inserts, updates and deletes. Some simple queries will be made easier as well.

By default, iBATIS 3.0 will be configured by convention. Java 5 method signatures have enough information to generate the SQL and map all parameters and results. For example:

Employee getEmployee (int id);

This is enough information to generate:

SELECT id, firstName, lastName FROM Employee WHERE id = ?

The result class is equal to the method return type. Because the method result class is not a collection, we know we're dealing with "SELECT one" semantics. The table name is assumed to be the same as the result class. The parameter is assumed to be ID for "SELECT one" methods (only because Java lacks introspection on parameter names...C# can do this). We can override the default ID parameter if we need to using one of the other configuration options (annotation, XML or Java API). The returned columns are assumed to be equal to the class properties but can be overridden too (see "Options" below).

Collections work the same way:

List<Employee> listAllEmployees ();

In this case we generate:

SELECT id, firstName, lastName FROM Employee

We can tell it's a "SELECT many" because the method returns a collection. We know the WHERE clause is empty because there are no parameters.

If we want parameters for a "query by example", we can support something like this:

List<Employee> findEmployeesLike(Employee employee);

Assuming only lastName is set, all other reference types are null, and primitives are set to some invalid number like -1...we can generate something like:

SELECT id, firstName, lastName FROM Employee WHERE lastName = 'Begin'

Or a better convention by naming could be:

List<Employee> findEmployeeByLastNameAndFirstName (String last, String first);

This will yeild:

SELECT id, firstName, lastName FROM Employee WHERE lastName = 'Begin' AND firstName = 'Clinton'

In C# this will be very nice, because C# can introspect on parameter names. It's quite a bit cleaner, but the same idea:

// C#
IList<Employee> FindEmployeesLike(string lastName, string firstName);

// And in C# 3.0, we can use anonymous types:

IList<Employee> FindEmployeesLike(object obj);
//...
IList<Employee> employees = mapper.FindEmployeesLike(new {LastName="Begin", FirstName="Clinton"});

Updates will work the same way:

void insertEmployee (Employee emp);
void updateEmployee (Employee emp); 
void deleteEmployee (Employee emp);

We can also accept collections automatically:

int insertEmployee (List<Employee> emps); // returns rows inserted
int updateEmployee (List<Employee> emps); // returns rows updated
int deleteEmployee (List<Employee> emps); // returns rows deleted

The generator could be smart enough to code a single statement for mass deletes (and maybe even some updates).

Relationships

We may be able to support some simple relationships. Simply having a complex property type or collection property type on a class is enough information to identify the relationship. Class definitions are descriptive enough to infer the relationship, even though the reverse is not true (i.e. the data model is not descriptive enough).

Standard SQL? What's that?

We're not going to go crazy with proprietary SQL support. In fact, we're not going to support it at all. We'll support the most standard SQL that works for most databases. This shouldn't be a problem considering the basic queries we're supporting in the convention based generation. Any crazy advanced SQL can be done by hand.

Some Possible Configuration Options

The following are some options we could support for SQL generation.

  • Assume that database follows JavaBean naming convention.
  • Translate JavaBean naming convention to UPPERCASE_UNDERSCORE database convention.
  • Primary key does (NOT) include table name.
  • Use SELECT * and lazy mapping for queries instead eager Class/Table mapping via Database Metadata at startup.
  • Use old WHERE clause join syntax
  • Default relationship loading to use join mapping or lazy loading

Configuration with Annotations

Annotations are becoming more popular and many people are choosing it over XML for metaprogramming. Configuration should not be included in annotations, but much of what iBATIS puts in XML is not configuration.

So what is configuration and what is not? Current iBATIS XML files contain 3 things:

  • Configuration
  • Meta information
  • Code

Configuration is anything that changes when you change environments. For example, database connection information, transaction management etc. In some cases if you're deploying to different databases and use proprietary SQL, then the configuration might include selecting the SQL dialect. We can include such things in iBATIS 3.0, so you can code multiple dialects (by hand of course), give them names and configure it in the appropriate location – outside of your .java files. Configuration should not be in .java files.

Meta information includes things like Result Maps, Parameter Maps and Cache Models. This information changes how iBATIS maps your data and how the mappings behave.

Code includes SQL and the dynamic SQL elemenets.

So what should be in annotations?

For the most part, only meta information should be in annotations. Configuration should be left to properties files or XML. Code should be left to Java code or XML. Furthermore, annotations should probably be used only for simple meta information. Annotations (especially in Java) can become complex, unreadable and messy.

So, I'd recommend using Annotations only as a means to override convention based configuration. That is, use it to tweak the odd SQL statement and the odd Result Map or Parameter Map that needs a column or two remapped.

Once again, C# Attributes will look a lot better than Java Annotations for a few reasons:

  • C# supports multiline strings, which makes inline SQL coding a lot more pleasant.
  • C# supports multiple attributes of the same type. Java requires horrendous collection attributes.
  • C# supports attributes with both ordinal and named parameters, making code much cleaner and more terse.

That said, the following is an example of how some of these annotations might look in Java.

NOTE: You'll notice also that I've included inline result syntax, as well as inline parameters. This topic will be controversial since the syntax for results look more like the old syntax for parameters. Overall though, I think this is a really significant improvement and an important one. Some people may choose to ignore inline parameters or results, or both, but many people will use them exclusively in favour of annotations or XML. Also as a sidenote, parameters will always be named from now on. No more "?" ever. The names can be referenced with "decorator annotations" to change properties of the parameter (e.g. types) etc.

// 
  // Simple select, string concatenation, inline results(!) and inline parameters
  //
  
  @Select("SELECT #id(EMP_ID:NUMERIC), #firstName(FIRST_NAME:VARCHAR), #lastName(LAST_NAME:VARCHAR) " +
      "FROM EMPLOYEE")
  List<Employee> selectAllEmployees();

  //
  // Alternative syntax using an array of strings instead of string concatenation...can be "smarter" than concatenation
  // 
  @Select({"SELECT #id(EMP_ID:NUMERIC), #firstName(FIRST_NAME:VARCHAR), #lastName(LAST_NAME:VARCHAR) ",
      "FROM EMPLOYEE",
      "WHERE EMP_ID = @id"})
  Employee selectEmployee(int id);

  // 
  // Inserts look as you might expect.  We can use getGeneratedKeys to get autogen key values, selectkey still supported
  //
  @Insert({"INSERT INTO EMPLOYEE (EMP_ID, FIRST_NAME, LAST_NAME)",
      "VALUES (@id, @firstName, @lastName)"})
  void insertEmployee(Employee emp);

  // 
  // Nothing special about update 
  //
  @Update({"UPDATE EMPLOYEE SET",
      "EMP_ID=@id(NUMERIC:IN), FIRST_NAME=@firstName(VARCHAR:IN), LAST_NAME=@lastName(VARCHAR:IN)"})
  void updateEmployee(Employee emp);

  // 
  // Delete is obvious.
  //
  @Delete("DELETE EMPLOYEE WHERE EMP_ID = @id")
  void deleteEmployee(int id);

The following examples are more complicated, almost to the point that this amount of annotation should probably be moved to the XML. I imagine we'll make the annotations full featured, in that anything that can be done in XML will be possible with annotations. However, it will likely be our running recommendation that XML be preferred for complex or even just verbose cases. Some people may choose to ignore the recommendation, and we'll support them in that.

// 
  // complex stuff
  //
  
  @ResultClass (Company.class)
  @ConstructorResults({
      @Result(property="id", column="C.COMP_ID"),
      @Result(property="name", column="C.NAME")
      })
  @PropertyResults({
      @Result(property="departments.id", column="D.DEPT_ID"),
      @Result(property="departments.name", column="D.NAME"),
      @Result(property="departments.employee.id", column="E.EMP_ID"),
      @Result(property="departments.employee.firstName", column="E.FIRST_NAME"),
      @Result(property="departments.employee.lastName", column="E.LAST_NAME")
      })
  @Collections ({
      @Collection(type=Department.class, property="departments", groupBy="id"),
      @Collection(type=Employee.class, property="departments.employees", groupBy="departments.id")
      })
  @Select("SELECT #id, #name, " +
          "#departments.id, #departments.name, " +
          "#departments.employees.id, #departments.employees.firstName, " +
          "#departments.employees.lastName " +
          "FROM COMPANY C, DEPARTMENT D, EMPLOYEE E " +
          "WHERE D.DEPT_ID = E.DEPT_ID " +
          "AND C.COMP_ID = D.COMP_ID")
  List<Company> selectAllCompaniesWithJoin();

  /*
   * NESTED QUERIES
   */

  @ResultClass (Company.class)
  @FieldResults({
      @Result(property="id", column="COMP_ID"),
      @Result(property="name", column="NAME")
      })
  @PropertyResults({
      @Result(property="departments",
          nestedQuery=@QueryMethod(type=CompanyMapper.class, methodName="getDepartmentsForCompany", parameters="id"))
      })
  @Select("SELECT #id, #name FROM COMPANY C ")
  List<Company> selectAllCompaniesWithNestedQueries();

  @ResultClass (Department.class)
  @PropertyResults({
      @Result(property="id", column="DEPT_ID"),
      @Result(property="name", column="NAME"),
      @Result(property="employees",
          nestedQuery=@QueryMethod(type=CompanyMapper.class, methodName="getEmployeesForDeparment", parameters="id"))
      })
  @Select("SELECT #id, #name FROM DEPARTMENT WHERE COMP_ID = @id ")
  List<Department> getDepartmentsForCompany(int id);

  @ResultClass (Employee.class)
  @PropertyResults({
      @Result(property="id", column="EMP_ID"),
      @Result(property="firstName", column="FIRST_NAME"),
      @Result(property="lastName", column="LAST_NAME")
      })
  @Select("SELECT #id, #firstName, #lastName FROM EMPLOYEE WHERE EMP_ID = @id ")
  List<Employee> getEmployeesForDepartment(int id);

Configuration with XML

XML configuration will be somewhat familiar, however there is a lot of room for improvement. Most notably, we'll support the following:

  • Result and Parameter mappings to fields, constructor parameters and JavaBeans properties.
    • Yet again, C# will rule with regard to constructor parameters, because they can be named. Java will be limited to ordinal constructor parameter mappings.
  • Join mapping and groupBy (N+1 selects solution) can be made much less verbose. We can reduce the need to create sub-resultmaps as well as make the syntax more clearly describe what the intent is (see Collection element below).
  • All Result and Parameter Maps will be "auto-mapped" so only columns that are mismatched against their properties need to be described
    • this dates back to a heated discussion from years back about ignoring missing columns, and/or auto-map extra columns
  • Improved/Simplified typehandler implementation and possibly data type conversion filters
  • The biggest change will be that the XML files will sit beside their Mapper.class counterparts. So EmployeeMapper.xml will be loaded for EmployeeMapper.class in the same classpath by calling something like config.addMapper(EmployeeMapper.class).

Important: People who wish to ignore the new convention and annotation based configuration can do so without missing out on any features. We're not changing the paradigm, we're adding to it. We may even allow people to set their configuration level by specifying "Convention", "Annotation", "XML" or "Java". That is, if they set it to XML, no convention or annotation configuration will be attempted. But even by setting Annotation, XML will still be attempted. So there's definitely a priority order to the configuration levels. Obviously disabling Java API configuration will not be possible.

<Mapper>

  <ResultMap id="selectACompanyWithJoin" resultClass="Company">
    <Constructor column="C.COMP_ID"/>
    <Constructor column="C.NAME"/>

    <Property name="departments.id" column="D.DEPT_ID"/>
    <Property name="departments.name" column="D.NAME"/>
    <Property name="departments.employee.id" column="E.EMP_ID"/>
    <Property name="departments.employee.firstName" column="E.FIRST_NAME"/>
    <Property name="departments.employee.lastName" column="E.LAST_NAME"/>

    <Collection type="Department.class" property="departments" groupBy="id"/>
    <Collection type="Employee.class" property="departments.employees" groupBy="departments.id"/>
  </ResultMap>

  <Select id="selectACompanyWithJoin" parameters="id:int,order:string">
      SELECT
        #{id},
        #{name},
        #{departments.id},
        #{departments.name},
        #{departments.employees.id},
        #{departments.employee{,
        #{departments.employees.lastName}
      FROM COMPANY C 
        INNER JOIN DEPARTMENT D ON C.COMP_ID = D.COMP_ID
        INNER JOIN EMPLOYEE E ON D.DEPT_ID = E.DEPT_ID
      WHERE
        C.COMP_ID = @{id}
      ORDER BY ${order}
  </Select>

</Mapper>

You'll notice a number of changes, especially with the select statement itself. The markup has changed to:

* #{column} is an inline result column, optional, and redundant in the above examle -- but 
ultimately should be able to describe most any column mapping situation just like an external 
result map.  Many people won't like this, but it will make others very happy.  Sorry, it will 
be a bit confusing that that # is the token used for results now, rather than parameters, but 
it makes more sense.

* @{property} is an inline parameter mapping to one of the provided parameters and its 
properties.  Much like inline parameters are today, very little difference other than the more 
common syntax using the @ symbol for parameters.

* ${property} is an inline substring replacement and is the inspiration for the new syntax and 
does not change at all really.  At most we may introduce simple SQL injection protection, 
possibly optional.

Configuration with Java API

Without going into details, all of the other configuration approaches will make use of a clean, test driven, configuration API that is suitable to release and support as a public API. The entire framework should be configurable from this Java API. In general, it will look something like this:

Reader configFile = Resources.loadAsReader("MapperConfig.xml"); //contains DB connection information etc.
Configuration config = new Configuration(configFile);
config.addMapper (EmployeeMapper.class); // Configures by convention, annotation, and from EmployeeMapper.xml in the same path.
//
//...more config of statements, result maps, parameter maps and cache contexts....syntax unknown...
//
MapperFactory factory = config.buildMapperFactory();
EmployeeMapper empMapper = factory.getMapper(EmployeeMapper.class); //generics will allow us to avoid the cast

Caching

  • Clinton says: Caching has never been very good in iBATIS...at least not as good as most ORMs, like TopLink and Hibernate. Unfortunately this is the reality with an SQL Mapping approach. Without introducing the concept of object identity and a number of other ORMish qualities, we can't support caching to the same level as the others. I don't believe caching is worth the loss of the core principles that make iBATIS different. So, if iBATIS caching is going to remain fairly basic, it should at least be easy. To date, cache configuration has been more work than it's worth. So, I believe we should make caching dead simple to configure, to achieve 80% of the value with 20% of the code. Any cases that require more advanced caching can write a custom cache above the persistence layer. For what it's worth, caching at the persistence layer is the least effective. Why? Because it's without user context. The best place to cache is at the web tier using a fragment caching approach that can understand more about user interactions and usage patterns that the persistence tier can never understand. In testing done with the Middleware Company years ago, we found business/context aware caches outperformed even the most advanced generic/holistic ORM caches every time.

So how is caching configured now?

As an annotation...

@CacheContext("Employee")
public class EmployeeMapper {
  void insertEmployee(Employee emp);
  void updateEmployee(Employee emp);
  void deleteEmployee(Employee emp);
  Employee getEmployee(int id);
  List<Employee> findEmployeesLike(Employee emp);
}

Or in XML...

<Mapper cacheContext="Employee">
  <Insert id="insertEmployee" ...> ... </Insert>
  <Update id="updateEmployee" ...> ... </Update>
  <Delete id="deleteEmployee" ...> ... </Delete>
  <Select id="getEmployee" ...> ... </Select>
  <Select id="findEmployeesLike" ...> ... </Select>
</Mapper>

The Cache Context basically links a bunch of mapped statements together. The results of any statements that query for objects and return either collections or single objects will be cached similar to the way they are today – but without the highly verbose configuration. Any mapped statements in the cache context that modify data (insert/update/delete) will flush the cache.

If there's a need to flush on some other statement (like a weird stored proc that updates and queries or something), it can be overridden using a simple annotation.

@flushCache
  List<Employee> updateAndGetSpecialEmployees();

Or XML...

<Select id="findEmployeesLike" flushCache="true"...> ... </Select>

Dynamic SQL

Thanks to Brandon Goodin, iBATIS has one of the most powerful dynamic SQL facilities around. There's nothing else quite like it. And now I'm about to suggest we completely change it.

iBATIS has always been heavily XML based. However, as it turns out, XML sucks for procedural logic. Unfortunately the Dynamic SQL as it is today is procedural logic expressed in XML. In addition to the simple fact that XML sucks at this, there are also a number of syntactic issues with special characters. Normally we handle special characters by simply wrapping the entire statement in a CDATA section. But with Dynamic SQL, we can end up with relevant XML that we need within the CDATA section. This creates a mess of procedural logic, SQL fragments and CDATA sections. While still better than a lot of alternatives, it's simply not the best we can do.

As I said before, I consider the Dynamic SQL facility to be "code" (as opposed to configuration or meta information). Code should be in a general purpose programming language like Java. So, what I will suggest is that we rebuild the Dynamic SQL facility as a really nice Java API that is every bit as good as the Dynamic SQL facility. This API can be use inside very simple Dynamic SQL Source classes.

Now that everyone is confused, let's use an example:

OldWay.xml
<select id="dynamicGetAccountList" parameterClass="Account" resultMap="account-result" >
  select * from ACCOUNT
    <dynamic prepend="WHERE">
      <isNotNull prepend="AND" property="FirstName">
        ( ACC_FIRST_NAME = #FirstName#
        <isNotNull prepend="OR" property="LastName">
          ACC_LAST_NAME = #LastName#
        </isNotNull>
        )
      </isNotNull>
      <isNotNull prepend="AND" property="EmailAddress">
        ACC_EMAIL like #EmailAddress#
      </isNotNull>
      <isGreaterThan prepend="AND" property="Id" compareValue="0">
        ACC_ID = #Id#
      </isGreaterThan>
    </dynamic>
  order by ACC_LAST_NAME
</select>
NewWay.java
// Warning: This is not well thought out.  There's a lot we can do with 
// anonymous inner classes, class initializers, generics and chained APIs (JMock)
// or even scripting languages.  This is simply for the sake of argument.
public class GetAccountListSQL extends SQLSource {
  public String getSQL(Object param) {
    Account acct = (Account) param;  // Generics might help eliminate this
    append("select * from ACCOUNT");
    prepend("WHERE"); //prepends before next append
    if (exists(acct.getEmailAddress())) {
      append("AND", "ACC_EMAIL like #EmailAddress#"); // append with prepend, first will be overridden
    }
    if (greaterThan(0,acct.getID())) {
      append("AND", "ACC_ID = #ID#");
    }
    prepend(); //clear prepend if any
    append ("order by ACCT_LAST_NAME");
  }
}

This SQLSource can then be used with an annotation:

@SQLSource(GetAccountListSQL.class)
List<Accout> getAccountList(Account acct);

Or through XML:

<Select id="getAccountList" source="org.apache.GetAccountListSQL" ...> ... </Select>

I think this approach is much cleaner and keeps code where code should be. It also makes it possible to easily introduce new types of SQLSources so that SQL can be generated by scripting languages or (if anyone is really tied to it) even XML...

XML Parameters and Results

We will continue to support primitives, POJOs, JavaBeans and Map types for parameters and results. XML support will also still exist, but will change dramatically. Instead of being supported as a first class type, we'll support it as a higher level type for all object types.

I was thinking of something as simple as introducing a standard iBATIS utility for serializing primitives, POJOs, JavaBeans and Maps. Or we could just tell people to use something like XStream.

Here's a ridiculously short example that should get the point accross.

return As.XML(empMapper.getEmployee(As.Integer("<value>5</value>")));

Hopefully you get the idea...basically XML is a higher level transformation to and from any object type. The benefit is that if you want an XML interface on top of your existing mapped statements, you don't have to remap them all to xml types...

Features Recommended for Exclusion from iBATIS 3.0

  • DAO (deprecated in 2.x)
  • PaginatedList (deprecated in 2.x)
  • NullValue (reason: it's ugly, confusing and pointless)
  • MaxSessions, MaxTransactions, MaxRequests (sessions and requests themselves may not exist in the new design, and TX configuration is the container's job)

Timeline

It's way too early for a timeline. The best we can do is guess at how long it will be before we have a Beta quality release. Perhaps below each committer can give a gut check:

Brandon:
Brice:
Clinton: 6 - 12 months
Gilles:
Jeff:
Jon:
Larry: 12-18 months
Nathan:
Roberto:
Ron:
Sven:
Ted:

JPA (iBATIS DataMapper)

In the section titled "Standard SQL? What's that?" I have an idea (well, OK, I stole it from the Hibernate guys and simplified it) that would make it easy for people using non-standard SQL. If we make the SQL generation done via an interface it can be pluggable. Then that section becomes:

===
We are not going to go crazy with proprietary SQL support. In fact, we are not going to support it at all.

We will support the most standard SQL that works for most databases. This shouldn't be a problem considering the basic queries we're supporting in the convention-based generation.

We will also provide a plug-in mechanism for users to extend that support to include vendor-specific SQL generation. Those plug-ins will most likely never become a supported part of iBATIS, but the authors and users are free to discuss them on the lists, or manage them however they see fit.

...and as always, any crazy advanced SQL can be done by hand.
===

I also think it would be great to have something concrete that describes the life cycle of a mapped statement's execution. Basically, what happens when and where, and ideally, each stage of that life cycle would be a separate pluggable component.

If someone wants to replace any part of the process, they should be able to do that easily, and at either globally or at the statement level.

IMO, that would make the code simpler (smaller blocks), and also facilitate the use of iBATIS for other types of query creation and data mapping (like the guy who was wanting to do LDAP data mapping).

>> I'm not opposed to the idea. However, I can't name a single
>> vendor specific feature that we could generate that would
>> warrant a pluggable generation mechanism.

I can: Oracle key generation is still manual, and since I use Oracle, it's important.

On another note, re: Transactions, another common question is "How do I roll back a transaction?", so I would like to see an explicit rollback() method on the transaction.

Transaction tx = factory.startTransaction(isolationLevel);
try {
  // do work
  tx.commit(); 
} catch(whatever) {
  tx.rollback();
} finally {
  tx.end();
}

Yes, I know it's redundant, but I think for most users, it's clearer than "end" meaning "end and rollback unless you have committed". At least for this user it is.

Larry

>> Oracle key generation is still manual,

You're right. Oracle will likely always be that way. To support generation of insert statements, we'll need that. (my original comment removed)

>> tx.rollback();

Argh. As much as I hate to do it, we might have to. I agree with the confusion, a lot of people ask about that. In addition to that there are problems with automatically calling (or not calling) rollback() with differences in drivers. For example, with Oracle, calling rollback is expensive, even if you didn't do anything (which is why we have the transactionRequired option). But with Sybase, NOT calling rollback, even after commit, causes the driver to be left in some weird inconsistent state. So I guess just leaving it up to the developers is the best thing.

I will say that I don't think we need both rollback() and end(). We should either use start/commit/end semantics, or start/commit/rollback.

Another option is to make rollback balk if nothing has been done within the transaction, so we could still do something like:

Transaction tx = factory.startTransaction(isolationLevel);
try {
  // do work
  tx.commit(); 
} finally {
  tx.rollbackIfNecessary();
}

In a sense, we'd just renaming end().

Good thoughts.

Clinton

Caching

"Any cases that require more advanced caching can write a custom cache above the persistence layer."

It sounds like we are forcing a closed option on caching rather than opening up the ability for people to "plugin" their own caching. To me it would seem to be a better option to provide a few good hooks that can help users to implement more custom caching rather than lock them out. This may even lead to some possible OID like caching strategies.

Some potential areas may include:

  • Providing access at the time an object is created
  • Expose primary key values pre object creation so that users can substitute an existing cached object into the results if they so desire.
  • Expose information about the sql map being called (id, resultclass, etc...)
  • Provide a way to simply expose all this to a class that implements a Cache interface (or something like). It would be up to them to handle the complicated internals. iBATIS would not be responsible for configuring it in any way. It just gives them information that could be valuable for them to hook in their own cache.

Dynamic SQL

I am completely on board with a Java based Dynamic SQL at the heart of any Dynamic SQL functionality. However, I do not believe the statements regarding XML "sucking" for this kind of use is accurate. IMO the XML markup can be less cluttered, less verbose, and more capable of expressing intention without losing any power. Most, if not all, of the dynamic SQL i have seen is not complicated by the fact that it was XML but rather that it was not a tight and simple set of tags. The nice thing about XML is that it is like being able to have easily nestable methods with optional parameters. Along with that a simple EL would allow for conditional decisions on inclusion of tag body content. You still get completion via a schema or DTD in any good xml editor. On top of this users may want to continue to keep their SQL organized nicely and cleanly together in an xml file. The only thing you would lose is completion on object properties. But, that is inherent in most all that we are doing. It is my opinion that retaining an implementation of the XML expressed dynamic SQL functionality as part of iBATIS is important and valuable.

On a related note, the two examples provided do not express the same SQL statement.

Dynamic SQL

While adding a Java based Dynamic SQL is an additional opportunity, I should consider XML as the preferred way of expressing complicated SQL code. The reason is that you get the SQL out of the code (which is always a good thing imho) and you can customize the sql with no need to touch the code.
Consider the following Oracle related problem
The same product is installed in two different sites and each site for table A generate very different data for column Col. If you have an histogram on column C you may modify the SQL where caluse to use a bind variable or a fixed value helping the CBO (Cost Base Optimizer) whether to choose an index or not. With the sql in the XML you can customize your product for every site, if you use code no.
I think it's worth mention that XML config files can be edited by DBAs while with Java code it's more difficult.
So I second Brandon opionion and suggest to find some way to improve Dynamic SQL functionality both in terms of power and usability

I think this proposal is a step in the right direction. While iBATIS is very clean and simple, there is certainly room for improvement. I'm particularly excited about the idea of having SQL generation based on convention, I think it'll be a time-saver feature.

Some comments/questions:

  • Will iBATIS 3.0 be Java 5.0 compatible only? If I understand correctly, some of the new features like Interface Binding will require features only available in Java 5.0. Will iBATIS 3.0 be still usable in Java 1.4, maybe without those features?
  • Interface Binding. The examples provided make use of only one method argument. What happens when there is more than one argument? Will iBATIS wrap them all in a Map automatically? How will the arguments be identified?
  • I like the idea of configuring iBATIS by convention. I agree with Clinton that it will be especially useful for inserts, updates and deletes. We (me and my workmates) rarely do anything fancy with those, so we'd happily let iBATIS generate them for us. However, I'd like to have some flexibility with regards to convention. For instance, on the examples posted it is assumed that the table name and the class name will match. I feel this is forcing users to use a certain database design convention, some users might want to pluralize table names (à la Ruby on Rails/Active Record) or add a prefix (TBL_). It could be easy to override those attributes via annotations or XML configuration, but I think it'd be a good idea to have a pluggable mechanism. Something like a Convention interface that could be implemented by the user. iBATIS would provide a default implementation (DefaultConvention?). This interface could define methods like getTableNameFor(Class clazz), getIdForSelectOne(Class clazz), getIdForSelectMany(Class clazz) etc. Users could implement all those methods or simply override the default implementation where necessary. I see this as a global custom convention mechanism.
  • I agree with Brandon. While I can see some people might prefer a less verbose Java based Dynamic SQL, I'm personally pretty happy with the XML based configuration system. It's definitely more verbose but I don't think it's that bad. It's probably a question of finding a better less verbose design, coupled with a simple EL for conditional decisions. Personally I believe the XML code is cleaner, it's easier to get lost on the Java based one. I'm afraid the Java based SQL would get messy with more complex queries. It's probably a question of taste, but both options should be available.

I must chime in on the dynamic SQL function too. Even though I understand and agree with the sentiment that XML is not a programming language, still I belive we must keep the dynamic SQL capability in XML. For me, the overriding consideration is that DBAs in large shops usually require an SQL review before going to production. If all SQL is in XML files, then it is easy to give the SQL/XML to the DBAs. If not, then there is a huge hassle involved with finding and assembling the SQL for review.

This basic idea of keeping the SQL separate from the code that executes it was the primary factor in my original decision to introduce iBATIS into my projects years ago. Anyone who works in a shop where there is an SQL review needs this kind of function. I think this simple architectural concept has made iBATIS as accepted as it is in large shops. I would hate to see it go - because I'd have to find another solution

But I do think that the dynamic SQL "language" could be improved. I'm in agreement with the old idea of making it look more like JSTL.

>> Will iBATIS 3.0 be Java 5.0 compatible only?

Interface binding can actually be done since 1.3 IIRC. It only required the dynamic proxy facilities. However, other features like annotation based config will require Java 5. My working assumption at this point is that we should just support Java 5. The two reasons being:

  1. By the time we have a first release, Java 5 will be 3 years old .
  2. iBATIS 2.x is still a very good solution for Java 1.4.

>> Interface Binding. The examples provided make use of only one method argument.

I've given this a lot of thought. We could support multiple parameters, but in Java (unfortunately) you'll need to name the parameters somehow, or use a messy ordinal notation. For example:

fetchEmployeeByFirstAndLastName (String firstName, String lastName);

Unfotunately in Java we lose the names of the parameters, so the best we could do is something like:

SELECT * FROM EMPLOYEE WHERE FNAME = @1 AND LNAME @2

In C#, the names of the parameters are preserved and can be accessed via reflection, so the SQL can make use of them like this:

SELECT * FROM EMPLOYEE WHERE FNAME = @firstName AND LNAME @lastName

So I'm almost inclined to leave it as a single parameter only. Then, if you want multiple params, you set them in a Map, just like you do today (and in C# 3.0 you can use an anonymous type too).

Oh how I wish Sun would shamelessly copy some C# features for the sake of the future of Java.

Cheers,
clinton

>> Interface Binding. The examples provided make use of only one method argument.

> I've given this a lot of thought. We could support multiple parameters, but
> in Java (unfortunately) you'll need to name the parameters somehow, or use
> a messy ordinal notation.

Brainstorming here..thinking as I type.

Could we use annotations to provide the names?

@ParameterNames("firstName", "lastName")
fetchEmployeeByFirstAndLastName (String firstName, String lastName);

Kind of redundant, because you have the names twice, but it should work OK, no?

Larry

>> @ParameterNames("firstName", "lastName")

Yeah, we could do that. We can default it to @1 and @2 unless they're overridden by the @ParameterNames annotation...at least until Java can reflect on method names.

The other way we could do it is by using the method name...

>> fetchEmployeeByFirstAndLastName (String firstName, String lastName);

Changing it to "fetchEmployeeByFirstNameAndLastName" lets us grab everything after the "By" and delimiit by "And" leaving only "FirstName" and "LastName".

Cheers,
Clinton

PS: In my last post it might not be clear that I'm suggesting doing all 3 approaches....

1) Look for the annotation, use it if there.
1b) C# can look at the parameter names and use those.
2) Look for a "By" clause in the method name.
3) Default to  (1,)2,@3.

It actually won't hurt anything to register the ordinal parameters and the named parameters so that you can use them interchangeably. This will avoid breaking existing statements if they change the name of the method to be more specific.

Clinton

I don't know what I typed in that last message, but apparently it's now a text field...thanks confluence...

Another feature I'd vote to put on the chopping block for i3 is lazy loading, which seems to be right up there at the top of the list of things that break frequently and we get questions about on the list.

IMO, it is a fragile obfuscation technique that adds no value in any situation that I can think of.

Let's consider an order/orderline type of thing...

In a web application, the request/response granularity is large enough that you need to know if you need the order lines at the same time that you need the order - because once the request is done, you can't go back and get it. Even with ajax you are sending another request.

In a desktop application, you reference to your dao layer to get the order anyway, so if you get an order then later need the order lines it's a method call either way (order.getOrderLineList() or orderDao.getOrderLineList(myOrder.getOrderId()), so there is not really any value added there.

My vote on "Let's give lazy-loading the axe." is +++1.

CONFIGURATION API

 Just a quick question on this:

Are we thinking of providing the ability to modify SQL statements at runtime via the API (without requiring server restarts etc)?

> Are we thinking of providing the ability to modify SQL statements
> at runtime via the API (without requiring server restarts etc)?

Yes, I think that is one goal of the new configuration API.

One thing that I am wondering though is how much will our ability to do this depend on the class loader? I am not convinced that there is a lot that we can do to make this happen if the class loader caches the resources.

>Yes, I think that is one goal of the new configuration API.

Great!

>One thing that I am wondering though is how much will our
> ability to do this depend on the class loader? I am not
> convinced that there is a lot that we can do to make this
> happen if the class loader caches the resources.

IMHO, I don't think we should be using classpath based resource loading to read SQLMap (XML) files in the first place. XML files should be read via standard java.io into configuration classes which then get stored as singletons or global lists.

Well, I think the way we do it (using the class loader) is probably the simplest way, but since we are exposing a public API, you are free to implement it any way you wish.

hi! I come from China, we have a project name "springside",I use iBatis 2 years,we like this have new feature support Annotations,and we plan extend iBatis 2.3,let it support dynamic SQL ,configuration with Annotations at our project.

<blockquote>Improved/Simplified typehandler implementation and possibly data type conversion filters</blockquote>

I suggest that TypeHandlers should be modal, e.g. could be passed enum constants from the xml. I find myself doing quite a lot of repetitive type handling, currently achieved with shallow subclasses to differentiate the functionality. It might look like this:

<result property="date" column="SSDDCURD"
		typeHandler="datetimeTypeHandler"
                typeMode="datetimeTypeHandler.MODE.Date"/>

	<result property="time" column="SSDDCURT"
		typeHandler="datetimeTypeHandler"
                typeMode="datetimeTypeHandler.MODE.Time"/>

iBATIS team.

This whiteboard really has great ideas and features. If implemented iBATIS would be a killer for Java 1.5+ apps.

I think there should be a link from the iBATIS homepage to this whiteboard. I only stumbled across it since I was reading a few posts from the developer archive.

We should add the Improved Dynamic SQL Whiteboard to this page/discussion. I think it's pretty clear that we want to retain the xml dynamic sql alongside the new dynamic capabilities.

I created a project called daozero(http://sourceforge.net/projects/dao-zero), just released version 0.5, and I find it's something similiar with interface binding.
I hope my thought below on dao-zero may be helpful with interface binding feature.
daozero is a spring factory bean that creates dynamic proxy bean to implement interface and call ibatis for interface.
In most cases, it just takes method name as ibatis statement name, and depends on method signature and statement meta data and a little conventions to decide to how to invoke ibatis.
For example,
1. multiple parameters could be combined into one parameter map. Currently I decide that method parameters have to keep the same occurance with statement parameters. I'm also trying to introduce annotation to elimite this limitation. And if for jdk 1.4, additional method signature along with parameter names is allowed to declare to work too. Also, additional method signature could be used to specify statement whose name is different with method name. I don't think redundancy here is problem because in mose cases keeping method name same as statement are enough and there're little occasion need for additional signature to work.
2. method sigature is also used to result transformation and parameter transformation and batch mode, etc. daozero can transform list result to array/set/iterator/other list; also can invoke ibatis for each element in parameters whose type is array/set/iterator/list, with batch mode option.

And I even also considered to support to generate statement by method convention and most of thought are same as ibatis, such as generate a "SELECT address FROM user WHERE name like ..." from a method "getAddressFromUserWithNameLike(...)", but I think it's not a good idea. it work for simple cases but unlike hibernate, there're no much accurate metadata for ibatis, so it will become complicated and error prone for more real case. Perhaps it is better to be an additional tool.

And for dynamic SQL, is following way suitable?
for
<select ...>
SQL "select * from user WHERE name=#name# AND address=#address#
</select>
if #address# is null or something else, truncate the SQL as "select * from user WHERE name=#name#" automatically. a additonal attribute "dynamic=true/false" could be used to declare whether it's dynamic statement.
Or another way is introduce special symbol. for example:
<select ...>
SQL "select * from user WHERE name=#name# AND address=@ address @
</select>
the @...@ means dynamic parameters and can be ignored if null, the "AND" before "address=@ address @" should be ignored automatically. additonal attribute "dynamic=true/false" is no need and become more flexible.

Posted by Yi-ting Qiang at May 14, 2007 10:47; last updated at May 14, 2007 21:33

Dynamic SQL
The main reason (really the only one...) that lead me to evaluate iBATIS and start to introduce it in my development group was that while letting java coders to do their job with objects it allows me to keep SQL code **OUT** of java classes.
We have legacy applications that deals with legacy DB's that contains tons of embedded, more or less dynamic, SQL that have become a nightmare to maintain.
iBATIS is THE SOLUTION because, as far as the result set structure does not change, it allows DBA's to maintain and optimize SQL as DB's evolve without touching a line of code.

So i'm not contrary to allow to manipulate or generate SQL from within java code as far as you continue to support FULL xml configuration for any kind of SQL code.
If i'd be forced to embed again SQL in java classes, then i'd have to develop an in house solution and quit iBATIS.

I re-read this list of comments, and I don't think anyone is suggesting that we eliminate the ability to use XML to describe your SQL statements.

What is being said is that you will be able to use iBATIS without as much (or possibly any) XML.