After a long conversation with Wade Wassenberg and reading his article about Protocol Independent Programming (PIP), I have decided to write a summary about my theory of software development.

There are tons of tools such as APIs and frameworks, which pretend to be an "everything what you need to build a successful project".

There are many guru voices: There cannot be THE tool (fw, api), which will solve all our problems. We should use several tools, APIs, frameworks to be successful.

So, what should one do? Think, and use paradigms or theories.

The following is "my" theory:

LAYERED APPLICATION DESIGN AND DEVELOPMENT (LADD).

From historical perspective and from my own experience I have made a conclusion:
No matter what kind of tools we are using, as soon as we are sticky with LADD, then the project is likely to be successful.

In short LADD is an application architecture, where system has clearly visible parts (layers) and every part is responsible for certain functionality. It is very important that there are calls only to next layer and no calls to skip a layer. There is no duplication of functionality. Object oriented design is one example of LADD. LADD is more than OOP, there are tons of examples outside of OOP like; mainframe architecture, UNIX, ANSI C standard, OSI stack of protocols and POSIX, to name a few.

Of course some tools and frameworks promote LADD and some have features, which create obstacles on the way to LADD.

LADD is not my invention, but purely a guidance, that could be found in many good books. I just realized how sticky I am to LADD and that sometimes people look to narrow and believe that certain feature may save the day.

To continue let consider "Patterns", all of them (Singleton, DAO, Factory, etc.), they are examples of demarcation between layers of functionality and responsibilities.

Lets consider a possible scenario:
We decided to use PIP (we can say JDBC, Axis, whatever we want here), it means that we have decided to code against a particular API. After using API across our system, we then realize that the API does not provide necessary functionality, or new major version comes to exist.

Now we have to go over the entire system to find and then fix all places where the API has been used.

What if we have coded our system in a manner, where everything related to deal with certain API (or functionality) has placed in one layer (could be library, class, package, etc.)? We know for sure where problems could happen and we will be able to fix them easier.

Concrete example could be: tons of SQL code across application versus placing such code in a specific layer, which is responsible for DB communication (DAO pattern). To continue the example: hard coded SQL code in DAO object or SQL in resource file. If I can see all SQL code in one place I will not worry too much whenever the code is SQL 92 compliant and portable across different RDBMS. Migration to another DB will be a relatively simple job.

With LADD in mind I consider PIP just as an API to use, and it is not a cornerstone. Communication protocol is important, but it is only a small part in a big and complex system infrastructure.

Let's discuss PIP from my standpoint. Value of PIP for business programming depends on availability of PIP framework(s) and suitability/maturity/completeness of such tool.

As PIP is very generic paradigm and has no connections with any business functionality, I would expect only software vendors investing time and money into development of tools to support PIP.

PIP is a very appealing idea and I know several implementations, which pretend to be a PIP:

SOAP is designed to be transport layer independent, that is, SOAP communications can happen over HTTP, SMTP, FTP, IP socket, something else. However SOAP imposes some limitations on business programming: one example might be a stateless nature of SOAP. So, if we need a session, we out of luck. (of course we can implement session tracking ourselves; or use vendor specific extensions; or wait until SOAP standard will include this feature ). Being immature technology it misses many important parts and standard language specific binding for instance. (In my opinion SOAP is all about reinventing CORBA, which we might see that as a way to sell CORBA to businesses. However, it puts so much trash in my head that I am not even glad I get paid for that.)

Of course if the paradigm behind SOAP PIP is sufficient, then we are in great shape.
Stateless Session Bean component inside of Java application server can be accessed via RMI, SOAP, or CORBA-IIOP and that takes close to 0(zero) efforts from Session Bean developer.

Another example might be Java (Java programming is platform independent because Java IS a platform). If we develop a business application, then Java's abilities are enough for us (?). But if we are going to develop high speed communication server (RDBMS for instance) then, Java will not be even considered as implementation platform. Even with the suitability of Java for business programming, nobody's business tries to tailor JVM, but participate in improvement the process via Java Community Process. One, maybe a negative example of development a JVM, is an approach that could be ORACLE Forms. From high perspective ORACLE Forms architecture it is roughly equal to JVM, but targeted to only one case and was developed by one vendor (do not forget, it is a software vendor with great experience in software development).

Every time we announce a new level of "independence" or "abstraction" we should be very careful.

Do we really define the level well?
Do we need to code against abstraction? Abstraction always comes with performance and flexibility price, not to mention extra efforts to create and maintain the abstraction level. ^comment
From which level to choose to start to code a particular project?

The entire IT industry is slowly developing such abstraction levels, starting from lower levels up to business layers. Let's name the layers:
Proc code -> assembler -> a language -> OS -> library -> standard library collection -> interpreter -> standard functionality ->...

Do you recall the time when nearly every company developed their own text editor, database server, HTTP-server and Servlet engine?

The speed of the process is really slow because software vendors are not very interested in the process.
Even most developers are not interested in that because of job security and a feeling, that developer's job becomes more and more resembles that "painting by numbers". Let's admit the fact: we are engineers, not scientists any more.

Because the process of improvements is driven by only the common sense, it will not be moving with lightning speed.

We have already discussed the broad perspective. Now, let me discuss the low level stuff, like Java Abstract classes and Interfaces that promotes LADD. (We should use them but...)

Abstract classes (abstract functions).
They do not have too much value. They are just a place to put common logic and a way to describe common functionality via abstract functions and to deliver such message via compiler. If tomorrow we wouldn't have abstract classes and functions, we could still achieve the same effect with real classes and empty functions. If we will mark such abstract functions with some Javadoc tag, then we can deliver our message to other developers via compiler in a fashion that is similar to a way of handling of @deprecated tag. "Abstract" - is just a way to implement LADD.

Java Interfaces.
They are a very important part of Java language that is difficult to deal with being open minded. Java interface is direct reflection of LADD in Java language, but, it is not only a way to code with respect to LADD. I-face is a message that says: "Do not worry what is behind this layer. You will always have the functionality that I describe".
Surprisingly, the same message will deliver a code, which uses "Reflection" or certain tricks like coding against common ancestors (abstract classes?).
What if there was a real class and no abstract parent and the class did not implement any interface? Does it mean we must NOT use API of such class as a boundary of a certain layer of functionality and must NOT code the rest of the application against the API?
I do not think so. Such class could be perfect implementation of a certain layer. What happened with the rest of an application if we would completely change class implementation and leave function signatures the same? The answer is: nothing.
We will recompile application and that is it. Such technique is widely used with code generators. Business delegate pattern is another way to have "Interface" without a formal one. In short Business delegate pattern means: we have a class that implements, DB access, and the rest of the application uses DB only through functions of the class.

Although I consider it to be good to code against Interfaces, even though I would admit, that in certain circumstances an attempt to bring formal interfaces could be harmful.

In general, there are no absolute rules. Even LADD is a bad idea when we need to get something done in 10 minutes.

Good judgment always should be used.

PS: Thank you Wade for valuable input.

PS2: Java critic

I value Java-J2EE for promoting of LADD, but Regexp functionality of String class in JDK 1.4 is disrespectful to LADD and makes me scary of Java future.
Yes, I am worrying about job security too....