Monday, December 29, 2008

Thoughts about SOA and beyond

Preface

Every serious framework has its Collection classes. Why? Because there is no application that does not need List or Hashtable, thus framework designers thought about that and provided a ready implementation, so we can focus on our application needs instead of writing our Lists and Hashtables over and over again.

Yep, thanks a lot. Now how many times have you defined your own fresh implementation for Customer or Order?

A word about SOA

SOA is about contracts. Of course, there is a famous ABC - Address-Binding-Contract, but "AB" are mostly "solved" by tools, meaning we don't waste time to provide them. Most of our time is spent on contracts, i.e. creating the required classes, getting the relevant data and filling it in, sometimes with other services and other times with our business logic and finally calling the required operation. Actually calling the operation is the 'finally' thing, the time was spent on creating the required classes, getting the relevant data and filling it in.

Why does it happen, why the tools do not help us? Because they do not know how! They 'see' those classes as property bags, they know what are the low level basic types of those properties, but they know nothing about their applicative meaning. So we spend best years of our lives to mapping between different types, yet very common in principle, but invented by us or different vendors.

Can you explain more?

Yes, of course. Frameworks did a great job in helping us to describe how my object is passed between the services. Well, thanks, but I also need some help to use that object. I mean that if I have a Person object and a Phone service, I want to be able to give this Phone service my Person and the tools will get the phone # from the person and pass it to the service. Or even more, I want to feed my grid with Persons and Phone service. The grid should understand that one of the Person fields is actually her phone #, render it as a link that when clicked calls the Phone service! And if this field is email and the service is Mail service, the click will open a 'new mail' window or start a chat or whatever, depending on the field and service in use.

As an application developer I need only to tell where are the 'Persons' and where are the services, all the rest should be done by tools, visual tools.

 

Wow, what do you need to get it live?

Actually much, but not too much. I need a common application object schema. This schema should define what is phone #, what is email and what is person that contains email, phone # and other information. This schema should not cover everything, it should provide only the common denominator around which the majority of our lives revolves. Just consider how much services we would be able to build/consume if we just had Person with its contacts (also Persons) with Address, email and may be some ID. Half of Google applications are built just out of this! And of course this schema must be publicly available, reasonable extensible etc, so every vendor will be able to extend it for his own needs; hopefully opening it for public too!

Interestingly that an effort to create this kind of schema was started, just did not get enough attention. It was a part of LDAP specification, see rfc4519 or rfc4524. Most of the examples I talked about can be created with elements defined there! Of course we don't have to evolve LDAP specification, but this can be a good start: standard concept, description language and even good, ready elements and many existing tools.

 

OK, you got your schema. Describe your perfect world!

Suppose we have a reasonable big number of common types.

  • There will be a substantially big number of services 'working' with those types. It will be clear to the vendors what API the world expects. The leading platforms, like .Net, will probably provide some basic services, like persistence, out of the box.
  • It will be much easier to consume those services, since they will fit each other.
  • Workflow engines and their complementary graphical designers will be able to manipulate on a much higher level of abstraction: there can be a concept of Person, Customer, Order etc as basic types with corresponding graphical representation. Probably many services will be also able to identify themselves and get correct glyphs.
  • GUI controls will also upgrade and operate with higher level objects. They will be also able to take service references as parameters to get their work done!
  • And finally every platform will provide a class library with those common types implemented and having many useful methods. Those types will be really first class citizens in the SOA world and application developers will need only extend them, if required. Their learning capabilities and efficiency will also increase since there will be much more common in this world!

 

This is how I would like it to be, and very interested to hear you opinion.

Please leave your comments!

Monday, July 7, 2008

Method Hiding - Feature or Bug in Compiler?

Since its very beginning C# has a 'method hiding' feature, which lets us 'hide' base class non-virtual methods. In other words one can write:
class Base
{
public void Test() {
Console.WriteLine("Base");
}
}

class Derived : Base
{
public new void Test() {
Console.WriteLine("Derived");
}
}

//...

Derived d = new Derived();
d.Test(); // "Derived" is printed


So far so good, but what happens if I write:



Base d = new Derived();
d.Test(); // "Base" is printed


Is this an intended behavior? Well, may be yes, but may be no. In case of 'no', we just called a wrong function, and no warning or error was issued by the compiler... Just consider again what happened, we declared a non-virtual method - that's our contract that should be ensured by the compiler. Instead it provides us with a feature to break it.



That's not all, consider the situation where you have some 3rd party library and use it in your project. Now the vendor releases a new version where he hides some method and provides an alternative implementation. Your program will still call the old method until you recompile! So in case you have several modules some recompiled and some not, you may have inconsistent behavior...



Of course you can always find use-cases, where the feature is carefully used and actually is very useful in that context. Yet its disadvantages outweight the advantages, so I think it would be better if we will develop our programs without using it.



Unfortunately there is an additional very similar "feature" - private interfaces, which can be implemented by a type even if its base class already implemented it and declared it sealed. Again the language helps us to break the contract and if it's not done carefully enough, the bugs won't wait to come...

Friday, July 4, 2008

C# Co[ntra]variance

To get introduced to the topic, please read Eric Lippert's variance series first.

Here I would like to present my solution to the problem.

After all there are only 3 possibilites:

  1. Clear covariance (IEnumerator<T>).
  2. Clear contravariance (IComparer<T>).
  3. Undecidable (IList<T>).

All the 3 possibility are easily detectable by compiler and I believe for the first two we would like the compiler will decide the variance automatically. We need a solution for the third case only.

To see the solution, I would like first analyze how we selected which case the given interface belongs to. We analyzed the method signatures in which T is involved:

  1. If in all of them T is 'out' parameter, the interface is covariant.
  2. If in all of them T is 'in' parameter, the interface is contravariant.
  3. If there is a mix - undecidable.

My solution is to say the compiler how I'm going to use my variable and enable either covariant or contravariant invocations:

IList<Animal> aa = whatever;
IList<in Giraffe> ag = aa;
IList<out Giraffe> agX = aa; //fails to compile

ag user sees: 'int ag.IndexOf(Giraffe)' and 'object ag[int]'.

IList<Giraffe> ag = whatever;
IList<out Animal> aa = ag;
IList<in Animal> aaX = ag; //fails to compile

aa user sees: 'int aa.IndexOf(<only null can be passed>)' and 'Animal aa[int]'.

Now I can say in a type safe manner:



class X<T> {

T _t;

void Test(IList<out T> at) {
T t = at[0]; //clearly t is 'out' parameter
}

void Test(IList<in T> at) { //overloaded method!
int i = at.IndexOf(_t); //clearly _t is 'in' parameter
}
}

//...

new X<Animal>().Test(new List<Giraffe>()); //first method is called
new X<Giraffe>().Test(new List<Animal>()); //second method is called

Monday, June 16, 2008

Domain Specific Language (DSL) with Workflow, Part 2

In the previous article I presented some basic ideas for creating DSL activities in Workflow. In this part I want to deep into the challenges I faced during the implementation and share the solutions. As usual, let's start from the requirements:

  1. DSL is a set of services, each one implemented by a separate function.
  2. For each such function provide a specialized Activity.
  3. All the return/out/ref parameters should work as expected.
  4. Developing of those Activities should be simple and provide Activity validation.

The first requirement is easy - I can have a separate method for each DSL service in my ExternalDataExchange interface:

// The ExternalDataExchange attribute is a required attribute 
// indicating that the local service participates in data exchange with a workflow
[ExternalDataExchange]
public interface ITaskService
{
int CreateTask(string taskId, string assignee, ref string text);

//...
}


To meet the next requirements I wanted to provide a base class, that by simple derivation from it and decorating my properties with some attribute, the desired DSL activity will be created:



[ToolboxItemAttribute(typeof(ActivityToolboxItem))]
public class CreateTask : DslActivity<CreateTask>
{
// Properties on the task
public static readonly DependencyProperty ReturnValueProperty = DependencyProperty.Register("ReturnValue", typeof(System.Int32), typeof(CreateTask));
public static readonly DependencyProperty AssigneeProperty = DependencyProperty.Register("Assignee", typeof(System.String), typeof(CreateTask));
public static readonly DependencyProperty TaskIdProperty = DependencyProperty.Register("TaskId", typeof(System.String), typeof(CreateTask));
public static readonly DependencyProperty TextProperty = DependencyProperty.Register("Text", typeof(System.String), typeof(CreateTask));

public CreateTask()
// Map the activity to the service method
: base(typeof(ITaskService), "CreateTask")
{
}

// Property, mapped to the return value of the method
[DesignerSerializationVisibilityAttribute(DesignerSerializationVisibility.Visible)]
[BrowsableAttribute(true)]
// "Well Known" name for the return value
[DslParameter(Name = "(ReturnValue)")]
[Category("ReturnValue")]
public int ReturnValue
{
get
{
return ((int)(base.GetValue(ReturnValueProperty)));
}
set
{
base.SetValue(ReturnValueProperty, value);
}
}

// Property, mapped to the 'assignee' parameter of the method
[DesignerSerializationVisibilityAttribute(DesignerSerializationVisibility.Visible)]
[BrowsableAttribute(true)]
[DslParameter(Name = "assignee")]
[Category("Parameters")]
public string Assignee
{
get
{
return ((string)(base.GetValue(AssigneeProperty)));
}
set
{
base.SetValue(AssigneeProperty, value);
}
}

// Other properties - mappings to the rest method parameters


In this way development of DSL activities is simple and straightforward. Now let's look into the base class to see how it's achieved. All the magic is done inside the constructor:



// Initializes parameters by reflecting the typeof(T) members
// and looking for DslParameterAttribute attribute.
private static readonly IList<ParameterInfo> _parameters = GetParameters(typeof(T));

public DslActivity(Type interfaceType, string methodName)
{
// Set up the interface and method
InterfaceType = interfaceType;
MethodName = methodName;

for (int i = 0; i < _parameters.Count; i++)
{
ParameterInfo pi = _parameters[i];
// This is the main 'trick' - create a binding to 'this' Activity
// with path set to the relevant property name
ActivityBind bind = new ActivityBind("/Self", pi.Path);
WorkflowParameterBinding binding = new WorkflowParameterBinding(pi.ParameterName);
binding.SetBinding(WorkflowParameterBinding.ValueProperty, bind);
ParameterBindings.Add(binding);
}
}


That's all, hope you enjoyed! Download here the sample code for this article.

Saturday, June 14, 2008

Domain Specific Language (DSL) with Workflow

Workflow provides two basic activities for interaction with outer world: CallExternalMethodActivity and HandleExternalEventActivity. But sometimes we want to limit our workflow to not use those, since they are too "generic". Instead, we want to provide our set of Domain Specific activities.

The simplest way to achieve this goal is by derivation, providing the required specialization.

Here is a link to a simple solution demonstrating the technique. In this example I create 2 application tasks and wait for completion of any of them. All the work is done with custom activities, defining my DSL.

Tuesday, June 3, 2008

Workflow (WF) RuleEngine without Workflow

Rule engine is a very useful feature and not always we want the entire Workflow to get into play.

Here is a simple code snippet showing how one can use RuleCondition/RuleSet outside the Workflow scope:

WorkflowMarkupSerializer serializer = new WorkflowMarkupSerializer();
using (Stream s = typeof(Program).Assembly.GetManifestResourceStream("SimplePolicyWorkflow.rules"))
{
RuleDefinitions rd = (RuleDefinitions)serializer.Deserialize(XmlReader.Create(s));
RuleCondition c = rd.Conditions[0]; // or RuleSet

//typeof(rule context) should be passed
RuleValidation v = new RuleValidation(typeof(Program), null);

if (c.Validate(v))
{
//rule context instance should be passed
RuleExecution exe = new RuleExecution(v, new Program());
c.Evaluate(exe);
}
}

Sunday, June 1, 2008

Self Code Review Methodology

When I worked as a team leader I didn't want any code to get into the product without the code review in addition to the unit tests added. But good code review takes time, so I started thinking how I can optimize the process...

First, I came to a conclusion that a review actually consists of two parts:

  • Code logic. This is best done by challenging. If the developer quickly and correctly answers to questions, he probably thought about that scenario and covered it. If not, we open the relevant code and check its logic in depth. This gives the opportunity to concentrate on less 'polished' code, analyze it deeper and correct more issues.
  • Code sanity, which includes:
    • Code format - ensured by IDE.
    • Code style - which actually is a routine check list. Bingo! Why not have this check list written and given to the developers to do themselves to save the reviewer's time?

Below is the check list I created, please leave your comments...

For each added field
  1. Consider type safety. I.e. when the field is used, no cast is required.
  2. Ensure its access modifier is private.
  3. Ensure its value cannot be computed by other means (using other fields/methods). If yes, it means this field is for caching purposes or used by some setter. If it's for caching:
    • Ensure this caching is required and comment this in code.
    • Ensure it is always in sync with computed value and comment how this is achieved in code.
  4. Consider marking this field 'readonly'. If not possible, consider refactoring resulting this field becomes 'readonly'.
For each added property
  1. Ensure it has a getter.
  2. Consider removing a setter.
  3. Ensure set/get parity, i.e. if some value is set, the same one is get.
  4. Ensure sequential gets return a same logical value.
For each added property/method
  1. Consider minimizing its access scope (private static <--> public virtual)
  2. Ensure all the arguments are checked and relevant exception are thrown.
For each added class
  1. Consider minimizing its access scope (private sealed inner class <--> public not sealed).
  2. Ensure it has a minimal set of constructors.
  3. Consider refactoring to have one constructor with initialization logic and other forwarding to it. If needed, add a private constructor.
  4. If there is an override for Equals/Hashcode methods, ensure they come in pair and both are computed from the same data.
For each local variable
  1. Consider type safety. I.e. when the variable is used, no cast is required.
  2. Minimize variable scope.
  3. Ensure proper clean up in finally blocks; usually initailization should be right before the try block and not inside it.
For each added line of code
  1. Consider implicit impacts (boxing, objects creation, computation).
  2. Ensure that every System.SystemException derived exception can be possibly thrown is by intention. (NullReference, Cast etc).
For each 'lock' statement
  1. Consider using a framework class, which has the required synchronization built-in. For example, if you need a synchronized Hashtable, don't do locking yourself, but create it with Hashtable.Synchronized() / SynchronizedKeyedCollection<K, T>.
  2. Otherwise ensure the design is discussed with your manager.

Wednesday, May 28, 2008

MVC Using Workflow (WF) as Model/Controller and ASP.NET as View

Download source code for this article
Agenda

Since workflow aims to provide an application business logic engine, it seems natural that we should be able to plug to it a presentation technology of our choice.

As usual, let's start from the requirements to our solution:
  1. Total separation between View and Workflow (Model/Controller) logic. We assume that Workflow is provided by an external team.
  2. Workflow should be developed with no assumptions on the presentation technology used.
  3. The components should be plugged by configuration.
Workflow definition

We assume it is an event driven state machine. This is how it looks for our example:

For each State we will create an interface with:

    1. Initialize() method, which is responsible for state initialization (in our case opening a form to collect data from the user).

    2. Several events that can happen in this state (user submits data).

    3. SubmitXXX methods, each of them raising a corresponding event.

    For example, for a State where we need to collect a customer name we will create a following interface:

    [ExternalDataExchange] //required for Workflow binding 
    public interface IGetNameState
    {
    event EventHandler NameReceived;

    void Initialize(Guid instanceId);
    void SubmitName(Guid instanceId, string first, string last); //raises NameReceived event.
    }

    At the end of the StateInitializationActivity we invoke the corresponding Initialize() method:


    to show the appropriate view.

    Once we are done for each State, we can provide our workflow to the Presentation team.

    Presentation

    First we must admit that the controller provided by the Workflow does not show any form. Further, it cannot do it by definition, since it does not know how! Instead it calls a relevant Initialize() method to be implemented by the concrete Controller.


    That means that we must provide a concrete Controller implementation to perform the actual navigation, i.e:

    void IGetNameState.Initialize(Guid instanceId)
    {
    HttpContext.Current.Server.Execute
    ("~/GetName.aspx?InstanceId=" + instanceId, false);
    }

    Later on we will see how we plug this Controller in by configuration.

    Now for each state the correct form is shown to the user, but his actions are not handled. Let's do it:

    protected void Button1_Click(object sender, EventArgs e)
    {
    WorkflowRuntime workflowRuntime = (WorkflowRuntime)Context.Application["WorkflowRuntime"];

    workflowRuntime.SubmitName(
    new Guid(Request.QueryString["InstanceId"]),
    txtFirstName.Text, txtLastName.Text);

    // Once we call Workflow, it's responsible to route the request
    Response.End();
    }
    Initialization and Configuration
    • Initialization:

      In the Application_Start handler we initialize the WorkflowRuntime:

      WorkflowRuntime workflowRuntime = new WorkflowRuntime("WorkflowRuntime"); //passing the configuration section name 

      workflowRuntime.StartRuntime();

      Application["WorkflowRuntime"] = workflowRuntime;

      and in Application_End we stop it:

      WorkflowRuntime workflowRuntime = (WorkflowRuntime)Application["WorkflowRuntime"];
      workflowRuntime.StopRuntime();
    • Configuration:

      Let's configure ExternalDataExchangeWorkflowService with our concrete Controller type:
      <ExternalDataExchangeWorkflowServices>
      <Services>
      <!-- The concrete Controller implementation -->
      <add type="Workflow.RuntimeServices.Controller, Controller, Version=1.0.0.0, Culture=neutral, PublicKeyToken=3a221384c77ffbe5"/>
      </Services>
      </ExternalDataExchangeWorkflowServices>
      <WorkflowRuntime>
      <Services>
      <add type="System.Workflow.Runtime.Hosting.ManualWorkflowSchedulerService, System.Workflow.Runtime, Version=3.0.00000.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
      <add type="System.Workflow.Activities.ExternalDataExchangeService, System.Workflow.Activities, Version=3.0.00000.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" ConfigurationSection="ExternalDataExchangeWorkflowServices"/>
      </Services>
      </WorkflowRuntime>
      Summary

      This tutorial helps to build a skeleton implementation of ASP.NET front-end for Workflow Controller. In our design we put a special attention on separation and correct interaction between the two. A very important property is that the Workflow does not dictate the implementation to the Presentation layer, its work can be performed in one or several steps. Only when all the information required by the Workflow is collected, it is invoked. For example, to collect the user full name, consisting of first and last names, the presentation may open one or two forms; provide any validation logic it wants and only at the end Submit the information. Such approach separates the Big application flow from the Presentation, opening a possibility to customize/update/change technology/upgrade it in any possible way.



    Saturday, May 24, 2008

    ASP.Net Unit Testing

    Some tell there are many solid solutions by HP (Mercury) or Microsoft's TeamSystem, some claim that all those are not Unit, but Integrative tests. And there are many that mostly perform manual testing... Who is right and what is the best approach?

    As usual, let's start from the requirements from the testing framework.

    1. Granularity. We are talking about unit testing, the name suggests that we want to test application units, separately.
    2. Ability to automate. Definitely we want an ability to run the tests automatically, for example after a build. This requirement also yields a requirement for results stability through the application life cycle.
    3. Coverage. Every part that is not covered by a unit test joins the risk group of deferred bugs. Even if there is integrative testing, it probably does not run frequently and therefore our control over correct system status reduces when the un-unit-tested group grows.

    Let's partition our application and see how we can test every part.

    1. Libraries, that do not require any "web context". Here we can simply write plain unit tests. No problems.
    2. Custom and User controls. In fact, those are also libraries, providing a well defined API, that can be systematically tested... But how to do it if they require "web context"?
    3. Pages, pages, pages, which actually produce our application on the server side and...
    4. Client side logic - let's not forget all the Java Script we run on the client browsers.

    Unfortunately most of the "well known" solutions, including those mentioned at the beginning address mainly point 3 above. The common pattern is to record the requests/responses sent/returned by the server; later sent those requests back to the servers, receive the response and validate it again previously recorded one.

    This introduces several problems:

    1. Granularity (since 'Custom and User controls' part is handled on the 'Page' level). When the test fails, we know that the Page failed, not some specific control and therefore had to invest more time in regression diagnostics.
    2. Ability to automate. Since the page evolves over time, our old recording may become invalid, even if everything is correct. This produces many false alarms, preventing automation.
    3. Coverage. Client side logic is left untested.

    In other words we have a problem in every aspect of our requirements, what can we do?

    1. Re-introducing test-driven development. Since not every code can be easily tested, we must include its testability into the very beginning of its design phase. The rule of dumb says that testability reduces when we move forward over the following path: 'Simple Library' -> 'Custom/User control' -> 'Page' -> 'Client Side logic'. This means that we must strive to move our application code as 'left' as possible.
    2. Create unit tests for our Custom/User controls. We use to create a separate test for every class in 'Simple Library', the same we can do with controls. Once we have a dedicated test page per control, it does not change when the application changes, improving our 'Ability to Automate' over time.
    3. Client Side logic testing. Since client logic runs in the browser, in order to test it, it must run in the browser, period. Any other option simply does not do the job. One of the best tools I know to run tests on the browser is Selenium - it will be covered in the next article.

    Summary

    1. Partition your application as described above.
    2. Try to move as much code as possible 'up' within the partition.
    3. For each User/Custom controls write a dedicated test page to test its functionality.
    4. Use some tool for testing client side logic, my recommendation is Selenium.
    5. If you go with Selenium for client side logic, for consistency use it also for Pages testing.