Workflow / Enterprise Service Bus (ESB)

[dev danke]

I'm working on a project to process product data. In some cases, the
data is processed in batch mode, such as when a supplier submits a new
list of products. In other cases, individual products need to be
reprocessed, such as when a business user manually changes the pricing
formula for a product in the system.

This system has both batch and ESB characteristics.

The current code is mainly in database stored procedures. But the
goal is to move this processing and business logic into Java. Dynamic
config data used in pricing rules will stay as data only in the DB.

There are several open-source solutions for these kinds of data
processing tasks. I'm looking at things like Apache Camel, Spring
Batch, Spring Integration etc. Does anyone have any positive or
negative experiences to share about working with any of these or
similar programs?

Thanks,
Dan


------------------------------------

Yahoo Groups Links

<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/seajug/

<*> Your email settings:
Individual Email | Traditional

<*> To change settings online go to:
http://groups.yahoo.com/group/seajug/join
(Yahoo! ID required)

<*> To change settings via email:
seajug-digest-***@public.gmane.org
seajug-fullfeatured-***@public.gmane.org

<*> To unsubscribe from this group, send an email to:
seajug-unsubscribe-***@public.gmane.org

<*> Your use of Yahoo Groups is subject to:
https://info.yahoo.com/legal/us/yahoo/utos/terms/

[Satish Viswanatham]

We use camel at ebates.com. We are happy with Camel. It's open source
community is strong. The main project contributors are talented and
accessible. We have never had a problem with Camel that We could not solve
myself, but if We did I know we can ask a question on the Camel forums and
the main dev will answer.

The level of satisfaction with Camel depends a lot on the mentality of the
developers using it. It is a bit like the wild west. There aren't
necessarily any best practices and any one thing can be done at least 3
different ways. That is why we like it :), but we can't guarantee others
will feel the same. Wedo like the huge number of Camel components that can
be used out of box.

There aren't that many integration frameworks out there. Camel and Mule are
the only ones. We don't have that much experience with Mule but I see that
most devs are using Camel. For a more formal approach look at Jboss Fuse.
This is just Camel in a pretty wrapper.

Let me know if you have any other questions.

thanks

I'm working on a project to process product data. In some cases, the
data is processed in batch mode, such as when a supplier submits a new
list of products. In other cases, individual products need to be
reprocessed, such as when a business user manually changes the pricing
formula for a product in the system.
This system has both batch and ESB characteristics.
The current code is mainly in database stored procedures. But the
goal is to move this processing and business logic into Java. Dynamic
config data used in pricing rules will stay as data only in the DB.
There are several open-source solutions for these kinds of data
processing tasks. I'm looking at things like Apache Camel, Spring
Batch, Spring Integration etc. Does anyone have any positive or
negative experiences to share about working with any of these or
similar programs?
Thanks,
Dan

[Satish Viswanatham]

We use camel at ebates.com. We are happy with Camel. It's open source
community is strong. The main project contributors are talented and
accessible. We have never had a problem with Camel that We could not solve
myself, but if We did I know we can ask a question on the Camel forums and
the main dev will answer.

The level of satisfaction with Camel depends a lot on the mentality of the
developers using it. It is a bit like the wild west. There aren't
necessarily any best practices and any one thing can be done at least 3
different ways. That is why we like it :), but we can't guarantee others
will feel the same. Wedo like the huge number of Camel components that can
be used out of box.

There aren't that many integration frameworks out there. Camel and Mule are
the only ones. We don't have that much experience with Mule but I see that
most devs are using Camel. For a more formal approach look at Jboss Fuse.
This is just Camel in a pretty wrapper.

Let me know if you have any other questions.

thanks

I'm working on a project to process product data. In some cases, the
data is processed in batch mode, such as when a supplier submits a new
list of products. In other cases, individual products need to be
reprocessed, such as when a business user manually changes the pricing
formula for a product in the system.
This system has both batch and ESB characteristics.
The current code is mainly in database stored procedures. But the
goal is to move this processing and business logic into Java. Dynamic
config data used in pricing rules will stay as data only in the DB.
There are several open-source solutions for these kinds of data
processing tasks. I'm looking at things like Apache Camel, Spring
Batch, Spring Integration etc. Does anyone have any positive or
negative experiences to share about working with any of these or
similar programs?
Thanks,
Dan

[Jason Osgood]

Hi Dan.

...project to process product data. In some cases, the data is processed in batch mode … ...project to process product data. In some cases, the data is processed in batch mode … In other cases, individual products need to be reprocessed,

Do the most simple thing that works. Adopt the unix strategy of composing many small work units. Each utility about 100-500 lines long. Each runs standalone and coordinates thru the file system. You already know what needs to be done. Don’t obfuscate the problem with some goofy framework.

---

ETL, workflow, ESB, queues, whatever or all euphemisms for 1) cutting a string from one place and pasting it somewhere else 2) logging the work.

Do not, under any circumstances, create or use IDLs, schemas, metadata models, mappings, etc.

It is mostly helpful to have a Composite class model, enabling fluent API esque, JXPath, Boon or other object graph navigation strategy.

I’ve used [*] eGate, ICAN, Microsoft BizTalk, Orion Rhapsody, Intersystems Ensemble, Talend, Apache Camel, ActiveMQ, and some ancient workflow engines from my CAD days. In fact, for a while I was the engineering manager for a “workflow engine” for the print manufacturing market. Input, processing, output with a pleasant UI.

To this day, I have no idea what a workflow engine is or why anyone would use one. The one possible exception are content management systems, where there are some human mediated approval steps.

What I do know is:

- “inbound" data, either push (e.g. HTTP server, JMS listener) or pull (FTP fetch, HTTP GET)

- “queues”, which is just some timestamped files in a directory, aka hotfolder. The fsync() method is your friend.

- “transformation” parse the data, maybe validate some stuff, change it somehow

- “outbound” data, write the data somewhere else, another queue, JDBC, email, HTTP POST, whatever

I created (with help) the ETL backend used in 5 regional healthcare exchanges. The easiest analogy: I designed the HL7 engine equivalent of the postfix email server (super simple, divide and conquer, composable, testable), whereas everyone else wrote SendMail (super complicated, hard to monitor, hard to extend).


I say all this (again) knowing full well that your team will pick some popular buzzword compliant off the shelf framework. Please remember, when your toes are curled over the ledge, your eyes downward cast into the abyss, despair filling your heart, please remember: I tried to save you.


Cheers, Jason


PS- Since we’re talking about Camel, I should express my gratitude. Camel cured me of my XSLT affinity. Choosing to use, nay — championing, Camel is in a 10 way tie for 1st place of the most stupid tech decision I’ve ever made.



[*] Experiences ranging from developing and supporting shipping products to proof of concepts.

[Satish Viswanatham]

well said Jason!

Do NOT use a framework unless you really really need to. But DO experiment
and try to not to reinvent the wheel too.

Post by Jason Osgood
Hi Dan.
...project to process product data. In some cases, the data is processed
in batch mode ... ...project to process product data. In some cases,
the data is processed in batch mode ... In other cases, individual products
need to be reprocessed,
Do the most simple thing that works. Adopt the unix strategy of composing
many small work units. Each utility about 100-500 lines long. Each runs
standalone and coordinates thru the file system. You already know what
needs to be done. Don't obfuscate the problem with some goofy framework.
---
ETL, workflow, ESB, queues, whatever or all euphemisms for 1) cutting a
string from one place and pasting it somewhere else 2) logging the work.
Do not, under any circumstances, create or use IDLs, schemas, metadata
models, mappings, etc.
It is mostly helpful to have a Composite class model, enabling fluent API
esque, JXPath, Boon or other object graph navigation strategy.
I've used [*] eGate, ICAN, Microsoft BizTalk, Orion Rhapsody, Intersystems
Ensemble, Talend, Apache Camel, ActiveMQ, and some ancient workflow engines
from my CAD days. In fact, for a while I was the engineering manager for a
"workflow engine" for the print manufacturing market. Input, processing,
output with a pleasant UI.
To this day, I have no idea what a workflow engine is or why anyone would
use one. The one possible exception are content management systems, where
there are some human mediated approval steps.
- "inbound" data, either push (e.g. HTTP server, JMS listener) or pull
(FTP fetch, HTTP GET)
- "queues", which is just some timestamped files in a directory, aka
hotfolder. The fsync() method is your friend.
- "transformation" parse the data, maybe validate some stuff, change it
somehow
- "outbound" data, write the data somewhere else, another queue, JDBC,
email, HTTP POST, whatever
I created (with help) the ETL backend used in 5 regional healthcare
exchanges. The easiest analogy: I designed the HL7 engine equivalent of the
postfix email server (super simple, divide and conquer, composable,
testable), whereas everyone else wrote SendMail (super complicated, hard to
monitor, hard to extend).
I say all this (again) knowing full well that your team will pick some
popular buzzword compliant off the shelf framework. Please remember, when
your toes are curled over the ledge, your eyes downward cast into the
abyss, despair filling your heart, please remember: I tried to save you.
Cheers, Jason
PS- Since we're talking about Camel, I should express my gratitude. Camel
cured me of my XSLT affinity. Choosing to use, nay -- championing, Camel is
in a 10 way tie for 1st place of the most stupid tech decision I've ever
made.
[*] Experiences ranging from developing and supporting shipping products
to proof of concepts.

[Nimret Sandhu]

+1!

Dan, is the work you're doing purely middleware or is it tied into the UI
layer at all (html, web 2.0, whatever)?

Most heterogeneous integrations I've done just exposed data via a UI
without a lot of business logic around that layer so I haven't really
needed to use any frameworks though I remember looking at Spring
integration at one point in time.

cheers,
-
Nimret Sandhu
http://www.nimret.org


On Tue, Mar 11, 2014 at 3:12 PM, Satish Viswanatham <

Post by Satish Viswanatham
well said Jason!
Do NOT use a framework unless you really really need to. But DO experiment
and try to not to reinvent the wheel too.

Post by Jason Osgood
Hi Dan.
...project to process product data. In some cases, the data is processed
in batch mode ... ...project to process product data. In some cases,
the data is processed in batch mode ... In other cases, individual
products need to be reprocessed,
Do the most simple thing that works. Adopt the unix strategy of composing
many small work units. Each utility about 100-500 lines long. Each runs
standalone and coordinates thru the file system. You already know what
needs to be done. Don't obfuscate the problem with some goofy framework.
---
ETL, workflow, ESB, queues, whatever or all euphemisms for 1) cutting a
string from one place and pasting it somewhere else 2) logging the work.
Do not, under any circumstances, create or use IDLs, schemas, metadata
models, mappings, etc.
It is mostly helpful to have a Composite class model, enabling fluent API
esque, JXPath, Boon or other object graph navigation strategy.
I've used [*] eGate, ICAN, Microsoft BizTalk, Orion Rhapsody,
Intersystems Ensemble, Talend, Apache Camel, ActiveMQ, and some ancient
workflow engines from my CAD days. In fact, for a while I was the
engineering manager for a "workflow engine" for the print manufacturing
market. Input, processing, output with a pleasant UI.
To this day, I have no idea what a workflow engine is or why anyone would
use one. The one possible exception are content management systems, where
there are some human mediated approval steps.
- "inbound" data, either push (e.g. HTTP server, JMS listener) or pull
(FTP fetch, HTTP GET)
- "queues", which is just some timestamped files in a directory, aka
hotfolder. The fsync() method is your friend.
- "transformation" parse the data, maybe validate some stuff, change it
somehow
- "outbound" data, write the data somewhere else, another queue, JDBC,
email, HTTP POST, whatever
I created (with help) the ETL backend used in 5 regional healthcare
exchanges. The easiest analogy: I designed the HL7 engine equivalent of the
postfix email server (super simple, divide and conquer, composable,
testable), whereas everyone else wrote SendMail (super complicated, hard to
monitor, hard to extend).
I say all this (again) knowing full well that your team will pick some
popular buzzword compliant off the shelf framework. Please remember, when
your toes are curled over the ledge, your eyes downward cast into the
abyss, despair filling your heart, please remember: I tried to save you.
Cheers, Jason
PS- Since we're talking about Camel, I should express my gratitude. Camel
cured me of my XSLT affinity. Choosing to use, nay -- championing, Camel is
in a 10 way tie for 1st place of the most stupid tech decision I've ever
made.
[*] Experiences ranging from developing and supporting shipping products
to proof of concepts.

--
thanks
Satish
I learn by Doing: http://www.linkedin.com/in/satishweb

[dev danke]

The workflow is strictly server-side data processing. Any UI is mainly for
business users to change business logic that affects how the product data
is handled.

An earlier comment said "...ETL, workflow, ESB, queues, whatever or all
euphemisms for 1) cutting a string from one place and pasting it somewhere
else 2) logging the work...".

I see many more things involved. There's error handling for one. How to
notify interested parties when things go wrong, and at what level of
granularity. For instance, when do you want to send an error for each item
versus waiting until the number of errors reaches a threshold and then send
a summary notification?

For batch processing, there are issues around defining, submitting,
starting, rejecting, stopping, resuming, and retrying a batch job etc.
There also needs to be ways of setting job priority. Some processing jobs
have more business value than others.

I once worked on a workflow oriented app where throttling was needed. One
part of the system generated data items faster than the downstream
end-point could absorb. I'd expect an ESB framework to have ways to handle
this situation.

Another issue is best-practices. Many frameworks try to use generally
accepted design patterns. This can be an advantage in a company with a
history of reinventing software patterns poorly and differently in each
part of the company. It leads to hard-to-maintain apps, that don't
integrate well, and become a drag on business innovation.

-Dan

Post by Nimret Sandhu
+1!
Dan, is the work you're doing purely middleware or is it tied into the UI
layer at all (html, web 2.0, whatever)?
Most heterogeneous integrations I've done just exposed data via a UI
without a lot of business logic around that layer so I haven't really
needed to use any frameworks though I remember looking at Spring
integration at one point in time.
cheers,
-
Nimret Sandhu
http://www.nimret.org
On Tue, Mar 11, 2014 at 3:12 PM, Satish Viswanatham <

Post by Satish Viswanatham
well said Jason!
Do NOT use a framework unless you really really need to. But DO
experiment and try to not to reinvent the wheel too.

Post by Jason Osgood
Hi Dan.
...project to process product data. In some cases, the data is processed
in batch mode ... ...project to process product data. In some cases,
the data is processed in batch mode ... In other cases, individual
products need to be reprocessed,
Do the most simple thing that works. Adopt the unix strategy of
composing many small work units. Each utility about 100-500 lines long.
Each runs standalone and coordinates thru the file system. You already know
what needs to be done. Don't obfuscate the problem with some goofy
framework.
---
ETL, workflow, ESB, queues, whatever or all euphemisms for 1) cutting a
string from one place and pasting it somewhere else 2) logging the work.
Do not, under any circumstances, create or use IDLs, schemas, metadata
models, mappings, etc.
It is mostly helpful to have a Composite class model, enabling fluent
API esque, JXPath, Boon or other object graph navigation strategy.
I've used [*] eGate, ICAN, Microsoft BizTalk, Orion Rhapsody,
Intersystems Ensemble, Talend, Apache Camel, ActiveMQ, and some ancient
workflow engines from my CAD days. In fact, for a while I was the
engineering manager for a "workflow engine" for the print manufacturing
market. Input, processing, output with a pleasant UI.
To this day, I have no idea what a workflow engine is or why anyone
would use one. The one possible exception are content management systems,
where there are some human mediated approval steps.
- "inbound" data, either push (e.g. HTTP server, JMS listener) or pull
(FTP fetch, HTTP GET)
- "queues", which is just some timestamped files in a directory, aka
hotfolder. The fsync() method is your friend.
- "transformation" parse the data, maybe validate some stuff, change it
somehow
- "outbound" data, write the data somewhere else, another queue, JDBC,
email, HTTP POST, whatever
I created (with help) the ETL backend used in 5 regional healthcare
exchanges. The easiest analogy: I designed the HL7 engine equivalent of the
postfix email server (super simple, divide and conquer, composable,
testable), whereas everyone else wrote SendMail (super complicated, hard to
monitor, hard to extend).
I say all this (again) knowing full well that your team will pick some
popular buzzword compliant off the shelf framework. Please remember, when
your toes are curled over the ledge, your eyes downward cast into the
abyss, despair filling your heart, please remember: I tried to save you.
Cheers, Jason
PS- Since we're talking about Camel, I should express my gratitude.
Camel cured me of my XSLT affinity. Choosing to use, nay -- championing,
Camel is in a 10 way tie for 1st place of the most stupid tech decision
I've ever made.
[*] Experiences ranging from developing and supporting shipping products
to proof of concepts.

--
thanks
Satish
I learn by Doing: http://www.linkedin.com/in/satishweb

[Jason Osgood]

Hi Dan.


My reading of your reply suggests you’re leaning towards a whizzy bang ESB, workflow engine, whatever. I won’t rehash my points, just add new ones.

I see many more things involved. There's error handling for one. How to notify interested parties when things go wrong, and at what level of granularity.

I reluctantly warmed up to the notion of using a separate monitoring tool for this. I had wanted to roll my own. Less is more, most “solutions” are turgidly overwrought, my ego, etc. Our support & ops people favored the familiar. For better or worse, they choose Nagios. Our little team put our efforts into tooling our services, to be better monitored. I now think leaning on a monitoring tool is the Correct Answer.

I once worked on a workflow oriented app where throttling was needed. One part of the system generated data items faster than the downstream end-point could absorb. I'd expect an ESB framework to have ways to handle this situation.

Hmmm. I favor pull over push. I cannot imagine a scenario where throttling should be done client side, but sometimes it must be. Must be a framework thing.

But this wish list item reminded me of something: My stuff could heal itself, with progressive back off. For example, if a necessary database was down, the feed would respectfully ping it periodically, patiently waiting for it to come back up. And then happily resume whatever work needed to be done. Worked brilliantly. I’ve not seen a “framework” which accounted for healing and robustness, much less do an awesome job. Though I did see some IBM whitepapers about “resilience”, so I’m sure it’ll be a hot new topic eventually. Hopefully someone will create a Spring plugin.

Until then, I expect more of the same. The Talend ETL work I just did was pretty typical. Unless the stars and moons are aligned just so, the “framework” dies a silent death and mocks your attempts to debug what happened.

Happy Hunting!


Cheers, Jason

[P.Hill]

Post by dev danke
For batch processing, there are issues around defining, submitting,
starting, rejecting, stopping, resuming, and retrying a batch job
etc. There also needs to be ways of setting job priority. Some
processing jobs have more business value than others.

For a basic framework or maybe just inspiration for how to create some
separation of concerns, you might take a look at Spring Batch.
http://docs.spring.io/spring-batch/2.1.x/index.html
It is one of the smaller applications of the Spring framework; thank
goodness. I used it on one project and found it pretty useful.

-Paul

[Paulo Avelar]

I have been using spring Integration, very happy with it. Using with AMQP
and rabbitmq. So far it rocks, almost no programming if you already have
spring beans.

Paulo

I'm working on a project to process product data. In some cases, the
data is processed in batch mode, such as when a supplier submits a new
list of products. In other cases, individual products need to be
reprocessed, such as when a business user manually changes the pricing
formula for a product in the system.
This system has both batch and ESB characteristics.
The current code is mainly in database stored procedures. But the
goal is to move this processing and business logic into Java. Dynamic
config data used in pricing rules will stay as data only in the DB.
There are several open-source solutions for these kinds of data
processing tasks. I'm looking at things like Apache Camel, Spring
Batch, Spring Integration etc. Does anyone have any positive or
negative experiences to share about working with any of these or
similar programs?
Thanks,
Dan