SQL access to your salesforce.com data enables standards-based data access for business intelligence, data warehousing, and ad-hoc reporting and analysis. Join us to learn how to achieve this with real-time SQL-92 access, using ODBC and JDBC. We’ll show examples of organizations using SQL access, and you’ll see how you can get even more value out of your business data with these techniques.
Image may be NSFW. Clik here to view.
Appropriate choice of shoe for the devzone
Session highlights:
Roar of the group discussion was muted at: 2:25
How it works: 6:00
Use Cases: 8:33
Real-time Data Visualization demo 11:07
Real-time Data Federation demo 16:50
Real-time Data Manipulation demo 22:18
Takeaways 27:50
Is that Chilly registering?
The data mining dog “Chilly” who built intelligence using SQL from force.com had a full conference pass waiting for him thanks to my developer evangelist and conference hosts!
Interested in using PHP on various platforms including Linux? Looking into integrating your website or application scripted in PHP with various data sources? You may want to check out our Connect for ODBC drivers.
The common problem faced by PHP developers is how to make a scalable and fast connection to various data sources. In today’s world, everyone uses the magic words: real-time integration. Using a standard-based third party driver can simplify the whole process, saving you money and developer time. If you choose a driver that is easy to white label, your customer doesn’t even need to know.
An Example: Connecting SQL Server on Linux via ODBC
Let’s say you are looking for a 64bit ODBC driver for SQL Server on Linux for a custom made application written in PHP. You need an easy solution to improve connectivity between the application and SQL server. Usually this means rebuilding the PHP code to establish the connection, but you already spent significant man-hours on the project, and who knows how many customers actually need the connection?
Well, an easy answer is that DataDirect Connect for ODBC provides connectivity from UNIX/Linux platforms to the Microsoft SQL Server. Since our drivers offer wire protocol solutions, you don’t need database vendor client libraries. So in short, not only is this a scalable solution, it will also be very fast one (I do think you have heard about the bulk load before…).
Just Imagine the Possibilities…
PHP is a robust scripting language useful for everything from small projects to enterprise application development. After all, the PHP motto is: PHP powers everything from your blog to the most popular websites in the world. By using DataDirect Connect for ODBC drivers, you can easily write portable PHP code for all major operating systems and platforms. DataDirect Connect for ODBC offers both performance and scalability benefits through the use of its unique wire protocol architecture. Wire protocol architecture also removes the need for vendor client libraries, which results in easier deployment and maintenance of your applications.
Oh, last but not least, DataDirect Connect for ODBC drivers are the most widely used third-party ODBC drivers in the software market today and are backed by an award-winning technical support team.
A Few Practical Points
Use PHP function odbc_connect()to establish a connection to a database. This function accepts three required and one optional argument:
resource odbc_connect (string dsn, string user, string password [, int cursor_type]) U S I N G P H P
The dsn argument must be a Data Source Name. The user and password arguments identify the user. Finally, the optional cursor_type argument is used to specify, if necessary, which cursor method to use. For more information about this argument and PHP ODBC functions, consult the PHP documentation.
On Windows, the ODBC Administrator, available through the Control Panel, provides a GUI for creating a data source that stores connection option values. On UNIX and Linux, you store data sources in a file called odbc.ini. Instructions for defining a data source can be found in the DataDirect Connect and Connect XE for ODBC User’s Guide and Reference.
If, for example, you created a data source named DataDirectMSSQL for connecting to a Microsoft SQL Server database, and had a user name of sa and a password admin, you would use the following syntax in PHP:
Following my third Dreamforce presentation on SQL data connectivity to Salesforce, I’m hearing more projects loading data into the Salesforce Platform. The sales ecosystem at Salesforce continues to grow and shifting data gravity to the platform for sales operations. With this shift, it’s making more and more sense to move data into the platform.
Why are shops moving data into Salesforce with SQL access?
On-premise Data Replication to the cloud to leverage unused capacity
Once the data is loaded into the platform, shops are leveraging the same DataDirect ODBC drivers to perform tasks such as real-time SQL data federation using SQL Server Linked Server and even some self service data connectivity using MS Access against Salesforce. While this all works seamlessly, there is a lot of work that went into making the driver efficiently write data into the platform.
Getting started loading data into SFDC using SSIS
Note: Screenshots are for SQL Server 2008 R2. Similar steps apply to SQL Server 2012.
2. Create a 32-bit and 64-bit ODBC System DSN to Salesforce.com with the SAME name (use shortcut from driver program group to load corresponding 32-bit or 64-bit ODBC Administrator). I used the name “Salesforce” as an example in screenshot below and specified a location to write the local schema files in the “Database” property to write to C:\ with file prefix “sumit.sandbox2″.
3. If you’re using SSIS 2012 or higher and inserting large numbers of records, check the “Enable Bulk Load” box on the bulk tab of the ODBC data source to transparently leverage Salesforce.com’s BULK API for optimal insertion. Large means “a few hundred thousand records” per Salesforce’s “Best Practices for Deployments with Large Data Volumes“.
5. Create new ADO.NET destination and select the Connection manager configured for Salesforce. For SSIS 2012, there are native ODBC sources and destinations which are pretty cool when you upgrade.
6. Map your source and destination columns. Note: The driver supports CREATE TABLE operations against Salesforce to create objects and fields in case you’re using replication or SQL tasks.
Following my widely popular tutorial for Business Objects Universe to Salesforce produced with SAP, I’m hearing several requests for real-time ODBC3 access to complementary data such as marketing automation in Eloqua, Marketo or Hubspot. In response to the feedback from numerous BOBJ shops and from BOBJ User Groups/Conferences – we are rapidly delivering these SaaS data sources for Business Objects shops via DataDirect Cloud.
Why love the Business Objects Universe to Marketing Automation
Marketing data is now sexy and multi source universe (MSU) support with BO 4.0 enables real-time data federation across sales data in Salesforce or Microsoft Dynamics with marketing data in Marketo, Eloqua and Hubspot.
This solution leverages existing skills and your investment in SAP. And Business Objects professionals are awesome, so try and network with them around the water cooler whenever possible.
Everyone is asking for it including the Eloqua topliners community, and ODBC3 access from DataDirect has been the unanimous choice for real-time Universe access to SaaS APIs.
DataDirect connectivity has the most secure architecture for accessing SaaS data from Business Objects. Most recently, we passed an IT security review consisting of 80 questions in flying colors.
(In the spirit of Universe technology, I want to emphasize that DataDirect Cloud is a real-time data service)
2. Select your marketing automation data source such as Eloqua, Marketo or Hubspot. In this case, I am selecting Eloqua since I have a production account.
Image may be NSFW. Clik here to view.
3. Configure your connection information to Eloqua. My Eloqua admin setup an account and I received an e-mail from support@eloqua.com with the connection information.
Image may be NSFW. Clik here to view.
I set user name, password and company using the custom properties as follows: “Company=Progress;FailOnIncompleteData=1″.
Note that the screen may change since this was taken during beta.
Image may be NSFW. Clik here to view.
4. Download and install the “Progress DataDirect Cloud Driver for ODBC” from DataDirect Cloud. For Universe Design Tool (UDT) or Information Design Tool (IDT), you will need a 32-bit ODBC client. For the Business Objects Application Server, you will need a 64-bit ODBC client.
5. Launch the ODBC Administrator from the “Progress DataDirect Cloud Driver for ODBC” program group and create a new data source. The Database Name corresponds to the name configured in step 3. You will login using your DataDirect Cloud ID from the ODBC administrator.
Image may be NSFW. Clik here to view.
6- Launch your design tool (IDT or UDT), and configure the Generic ODBC 3 data source as type, “ODBC3?.
7- Enter User name, Password and Data source name as “Eloqua” in this example.
8- Click Next and continue to design your Universe to Eloqua similar to any other database.
Image may be NSFW. Clik here to view.
Enjoy the all access data pass from your Universe
If you want to add MSU connectivity to SaaS data sources such as Salesforce, Microsoft Dynamics, Marketo or Hubspot: simply login to your DataDirect Cloud account and create a new connection. Then create a new local ODBC data source using the same Progress DataDirect Cloud Driver for ODBC that points to the new connection.
Since these are standard ODBC compliant connections, you can also use them from Data Services running on either Windows or Unix/Linux.
Call 1-800-876-3101 to speak with a systems engineer and learn how other organizations are making progress after creating a BOBJ Universe to marketing automation data.
There is a growing sentiment among data connectivity professionals, in the real world, around disruption. From my consulting projects, it typically starts out with a requirement to integrate a new system and we deliver the data connectivity.
Next, we have a follow-up call with an Infrastructure Team Lead, Data Architect, or CDO/CIO; and they start telling me about the importance of standardization on DataDirect Data Connectivity. I initially thought these were prank calls by our marketing folks that say the same thing, but in reality it’s just a sign of our business becoming more relevant than ever before.
Let there be disruption
It’s true, Progress Software is also guilty of disruption with our new cloud platform, Progress Pacific, that includes aPaaS. But we’re also guilty of disrupting our own disruption (which makes everyone happy) by providing real-time direct ODBC/JDBC SQL access to the applications built on the Rollbase aPaaS, as well as standards based data access to external sources – both directions made possible by DataDirect Cloud.
But it gets really ugly outside the zen of Progress Software. We see data silos in SaaS applications (i.e. Salesforce,Dynamics,Maketo), NoSQL (i.e. MongoDB or Cassandra); IoT (i.e. machine data); NewSQL; Graph DBs; in-Memory DBs; and of course big data. Each of these disruptive technologies brings tremendous potential and ROI if we can just avoid the disruption of data integration.
Enable the data professionals to disrupt the disruption
Let me share concrete examples related to disruptive technologies and data connectivity from real-world projects on the ground:
1. Application Platform As A Service (aPaaS) has arrived
I have worked on several projects connecting existing data infrastructure to aPaaS platforms such as Force.com and Rollbase. One common use cases is for SAP Business Objects Universe to Salesforce.com data. The existing SAP BI infrastructure expects real-time SQL access to an operational data store, however many aPaaS platforms create a data silo where the database is inaccessible behind a public or private cloud. In fact we’ve been told numerous times, which I shared at Dreamforce, that DataDirect connectivity enables aPaaS adoption when organizations require operational intelligence and real-time data access. The aPaaS market is growing, and standard data connectivity to SaaS/aPaaS from DataDirect is here to disrupt that disruption for your existing systems.
Image may be NSFW. Clik here to view.
2. Disruptive new system getting dumped on your team
I’m hearing this from data architects, CDOs, or CIOs from verticals such as Financial Services to Telco. The challenge is the variety of specialized data sources coming in such as IoT data in compressed formats, SaaS applications, RDBMS with machine data in Postgres or market data in Sybase, NoSQL or NewSQL databases running fraud detection, etc. Rather than hire a staff of data scientists, java developers, curl developers, etc to access data; leaders in this space are partnering with DataDirect (and our partners) to put a SQL front end on the disparate data to hit the ground running with those disruptive technologies from end to end.
Standardization from a development standpoint has been solved via ODBC and JDBC (thanks in large part to DataDirect’s professional contributions). Since SQL can still vary, one approach has been to pick a database technology (so far Oracle or SQL Server); and connect to the disruptive world through DataDirect ODBC drivers. The SQL Server Linked Server or Oracle Database Gateway technologies combined with DataDirect’s breadth of data sources provide that standard SQL front end to the disruption.
As requirements grow, DataDirect standards based connectivity is flexible for expansion outside the RDBMS technology layer for direct connectivity from ETL, data warehousing, BI, analytics, application development, etc. It makes a great addition to corporate application stores and marketplaces.
3. ISVs with disruptive technologies
If you’re an ISV building a solution without direct SQL access to the data, please consider building an ODBC drivers for all my friends in the enterprise. I really love how Netsuite provided ODBC access to their SaaS application; and how Progress Software provided standards based connectivity to Rollbase on the Progress Pacific aPaaS. We’re already hearing from Rollbase shops that this clears adoption hurdles for data driven organizations running Microsoft Excel/Access and BI/Data Visualization Tools.
Disrupt your day to get started disrupting the disruption
2. Or call 1-800-876-3101 to speak live with a Systems Engineer to learn more. We promise to limit saying the increasingly overused word “disruption” to 3 times on the call.
Based on some recent projects, I am seeing a trend that continues to blur the lines between business intelligence (BI) and application development. It’s only fitting to blog about this topic since Progress Software is the worldwide leader in data connectivity and earned Gartner’s Visionary status for its application development platform.
My recent intelligent application consulting projects:
BI Platform
Requirements
Sources of Data
Oracle Business Intelligence OBIEE
Dashboards that enable core interactive business decisions in real-time
Oracle, Force.com
SAP Business Objects
Intelligence portal that merges data warehouse history with up to the minute search capabilities
SQL Server, Apache SOLR
Qlikview
Interactive charts to move historic opportunity data from warehouse to operational CRM
Teradata, Salesforce
Tableau hosted in Rollbase
Interactive project activity application that refreshes analytics in real-time using DataDirect Cloud.
Rollbase, OpenEdge
As the appdev/BI lines continue to blur, there is increased interest in querying unstructured data such as Apache SOLR. To provide standards based connectivity to Solr, DataDirect provides self service rapid development capabilities to build a prototype in days. This is achieved with DataDirect OpenAccess, and I will share my Apache Solr ODBC connector built last Saturday morning that took me less than an hour.
6- Once complete, below is sample query using Interactive SQL (ODBC):
ISQL> SELECT * FROM SOLRDOCS WHERE QUERY = ‘solr’
NAME ID FEATURES QUERY
Solr, the Enterprise Search Server SOLR1000 {id=SOLR1000, name=Solr,
the Enterprise Search Server, manu=Apache Software Foundation, cat=[software, s
earch], features=[Advanced Full-Text Search Capabilities using Lucene, Optimized
for High Volume Web Traffic, Standards Based Open Interfaces - XML and HTTP, Co
mprehensive HTML Administration Interfaces, Scalability - Efficient Replication
to other Solr Search Servers, Flexible and Adaptable with XML configuration and
Schema, Good unicode support: héllo (hello with an accent over the e)], price=0.
0, price_c=0,USD, popularity=10, inStock=true, incubationdate_dt=Mon Jan 16 19:0
0:00 EST 2006, _version_=1457970426419347456} *
Are you a data architect, project manager, or java developer with questions?
Call us at 1-800-876-3101 to learn more about how DataDirect OpenAccess can get you connected using ODBC or JDBC to Apache SOLR, as well as any web service API or unstructured data source. This is one of those projects everyone at a data driven organization will want their hands on.
From working on data connectivity projects for a decade, I can say with certainty that hybrid cloud / on-premise environments have become the norm. And I’m hearing requirements for secure data connectivity to data stores that reside on-premise behind a fire wall; or in a public cloud server running on Amazon. Rather than open a database port, VPN, or SSH tunneling; you can expose that data using the DataDirect Cloud On-Premise Connector with real-time and direct ODBC/JDBC data connectivity. Today, we introduced support for OpenEdge 10.2B to 11.3, and Oracle, SQL Server and DB2 are coming soon using the same proven approach.
User Tested, CIO Approved
The DataDirect Cloud On-Premise connector capability certainly raises questions for the on-premise world, especially in very large enterprises. Last week, I demo’d the on-premise connector technology to Progress Software’s CIO, Dave Benson, and his team. After addressing their security questions, the team saw the benefits with some current initiatives at Progress Software, and approved of the architecture. Progress IT recommends partnering IT with business sponsors for collaboration on data connectivity projects in case there are concerns around access to an IT managed database.
Prezi for ODBC/JDBC from SaaS/PaaS over HTTPS to on-premise OpenEdge, Oracle, SQL Server, DB2
Highlighting capabilities of DataDirect Cloud and Rollbase
The prezi for my Progress Communities Webinar is followed by two live demos. First, I show the hosted version of Rollbase federating external objects directly from on-premise RDBMS, the ubiquitous OpenEdge sports database. Then, we can built a chart of items in the hosted SaaS BI application, Explore Analytics. Finally, to prove the direct real-time connectivity to on-premise; we update a federated record from Rollbase and re-run the charts in Explore Analytics to ensure the updated record is coming back. At this point, attendees will be mesmerized by the data connectivity!
Infosphere DataStage is one of the top data integration platforms from a data connectivity perspective since most sources and targets are fueled by DataDirect ODBC connectivity. This includes our streaming bulk load (without landing flat files) functionality for faster loads into popular RDBMS such as SQL Server from Unix/Linux.
I have been working with an increasing number of DataStage shops looking for expanded connectivity to SaaS applications using flexible ODBC access direct to Salesforce, Veeva CRM, Eloqua, Oracle RightNow and Marketo. DataDirect delivers SaaS data connectivity that is built from the same framework as relational ODBC drivers already bundled with DataStage today.
Why add DataDirect SaaS connectivity to your stage?
We’re putting a SQL-92 ODBC compliant interface on SaaS applications so ETL developers can hit the ground running. You don’t even have to know how to spell “eloqua”.
Disruptive SaaS data sources are coming soon to a DataStage near you. I’ve seen a couple projects where organizations are forced to use a kludgy architecture to consume SaaS data against your own best practices/data governance for consuming SaaS data.
IT can take control of the SaaS landscape before business sponsors even know what to ask. It’s really important to me that you use the best platform for the job without spinning up one-off satellite data integration systems.
The same connectivity works with any standards based IBM application including IBM Cognos (Data Manager, BI, Transformer), IBM Cognos TM1, IBM SPSS, DB2 Federation Server, Websphere Application Server … as well as any standards based applications from Oracle, SAP, or Microsoft.
Getting Started unleashing the power of DataStage on SaaS
Below are steps to integrate additional DataDirect ODBC drivers with DataStage:
Verify test-connection works independently from DataStage by following the corresponding quick start guide with the download.
Integrate the ODBC shared library path (e.g. LIBPATH on AIX, SHLIB_PATH on HP-UX pa-risc, and LD_LIBRARY_PATH on Solaris, Linux and HP-UX Itanium) in <InformationServer>/DSEngine/dsenv file. (Note: If running on Websphere, the shared library path will need to be configured from Websphere)
Specify the ODBC data source name in the uvodbc.config
Merge the odbc.ini data source into the existing file used by DataStage typically found in $DSHOME. Below is a sample Salesforce DSN:
[Salesforce]
Driver=/opt/Progress/DataDirect/Connect64_for_ODBC_71/lib/ddsfrc27.so
Description=DataDirect 7.1 Salesforce
HostName=login.salesforce.com
SecurityToken=<your security token>
StmtCallLimit=0
TransactionMode=1
Database=/tmp/sfdc
Optional: The Connect XE Salesforce.com driver requires a JRE, and if DataStage is running with an older JRE version, the driver can be configured to use a standalone JRE independent from DataStage by following these steps:
- Configure shared library path with desired JRE. For example on AIX: - LIBPATH=/usr/java6_64/jre/lib/ppc64:/usr/java6_64/jre/lib/ppc64/classic:$LIBPATH
- Add following property to Salesforce ODBC DSN in odbc.ini file: SQLEngineMode=1
- Check whether the standalone java service is running, ps -ef | grep “sforce”
- If not running, please start it from the following directory: /opt/Progress/DataDirect/Connect64_for_ODBC_71/java/lib
- java -cp sforce.jar com.ddtek.cloud.sql.Server &
Restart the node agents and begin importing SaaS metadata via ODBC
Image may be NSFW. Clik here to view.
Information Server Designer Client ODBC connection to SaaS
Learn more about the expanded Data Connectivity Options available from DataDirect
2. Or call 1-800-876-3101 to speak with a systems engineer to learn how other organizations are making progress with DataDirect Connect Salesforce integration.
A few weeks back a colleague and I were driving through Seattle, and we began to debate the reach of ODBC as a key data standard. His perspective was that ODBC was really a C/C++ technology, and that as people have largely moved to other programming languages, the importance of ODBC has declined as well. Well, I didn’t buy that argument but needed to do some research before I could refute him. As I mainly tinker with iOS development these days, my awareness of what is happening in the programming world at large is well out-of-date.
Well, it didn’t take long to dig up some examples:
ODBC for PHP. Basically, as long as you have an ODBC driver, PHP will support it just lovely. Here is the manual on how to configure it: http://us2.php.net/manual/en/book.uodbc.php
ODBC for Erlang. Well, that is pretty cool. Trying to write a highly parallelized application that deals with a lot of data? Can’t get much better of a combo than this! Here’s the manual pages that show how to set this up too: http://www.erlang.org/doc/man/odbc.html
ODBC for Python. Okay, I think I’m starting to prove my point. Here’s the link to setting this up: https://wiki.python.org/moin/ODBC
What is interesting is that most of the examples here refer to relational databases. But in case you didn’t already know, what Progress DataDirect also does really well is make any cloud application and Big Data, NoSQL database look and feel just like a relational database. So let’s say you want to do things like:
For “big data” connectivity, I see a very high percentage of projects delivering business value from big data initiatives that require data connectivity from DataDirect. This was specifically around providing Universal data connectivity to Hadoop Hive and related low latency interfaces (Impala, Hawq, Drill, Stinger, etc).
From recent active projects, we’re connecting systems in the other direction which takes success to new levels; and that is when you can move data sets aggregated by the Hadoop big data ecosystem; and move them to relational databases managed by strict corporate security policies which means your data is above the radar now. The most common is the requirement to connect to SQL Server from Linux using Windows Authentication (example from Hortonworks community).
How ODBC and JDBC SQL Server Windows Authentication from Linux works:
DataDirect is your exclusive partner in data connectivity providing unique support for Windows Authentication from Linux across both ODBC and JDBC connectivity. Sqoop is a JDBC utility to move data between Hadoop Ecosystems and relational databases such as SQL Server. On the ODBC side, we’re seeing Python/Ruby wrappers that need to connect to relational DBs. Microsoft recommends using integrated security over mixed mode authentication for secure SQL Server platforms.
DataDirect invented “wire protocol” technology to connect to data without data source specific dependencies; and our implementation for windows authentication from Unix/Linux is no exception. You simply need to configure the driver with your Windows credentials in plain text. For Kerberos support, no credentials would be required.
Getting started with ODBC and JDBC SQL Server NTLM authentication from Linux:
Not even the sexiest data scientist and Hadoop platform are above corporate security, so here’s how you can connect from JDBC and ODBC to SQL Server to remain in compliance:
[SQLServer]
Driver=/opt/Progress/DataDirect/Connect64_for_ODBC_71/lib/ddsqls27.so
Description=DataDirect 7.1 SQL Server Wire Protocol
AuthenticationMethod=9
Database=test
Domain=AMERICAS
EnableBulkLoad=0
#Connecting to SQL Server Named Instance
HostName=172.30.57.75\SQL2008R2SP2
LoadBalancing=0
Pooling=0
# PortNumber not required when connecting to Named Instance
#PortNumber=1433
During test connect with the DataDirect samples/example utility, specify the following credentials for windows ntlm authenticated user “AMERICAS\SUMITS” and note domain is not required as part of user name since it was configured in the data source:
sumits@nc-lnx16 /opt/Progress/DataDirect/Connect64_for_ODBC_71/samples/example> example
example DataDirect Technologies, Inc. ODBC Example Application.
Enter the data source name : topptt01_cs
Enter the user name : SUMITS
Enter the password : my_windows_auth_pwd
Enter SQL statements (Press ENTER to QUIT)
SQL>
Download the DataDirect Connect for JDBC drivers for and follow the quick start guides supplied with the download. The SQL Server JDBC connection URL should be configured as follows for NTLM authentication:
Oracle TimesTen is an in-memory database that can be used as the core database of record or as a cache database for Oracle. When used as a core database for data intensive workloads, we’re seeing significant demand to setup an Oracle Database Gateway to federate TimesTen data for downstream applications, including those built on Oracle.
Downstream with Oracle TimesTen without a paddle
If you search Oracle Support, you’ll find that the Oracle TimesTen client by itself is not supported with the Oracle Database Gateway for ODBC per Oracle Support, “Connecting Oracle to Timesten Using DG4ODBC in Linux x86 and x86-64 (Doc ID 1444607.1)”. This is because the Oracle Database Gateway for ODBC (DG4ODBC) 11g is ODBC V3 while TimesTen ODBC-Driver/Manager only supports ODBC V2 calls.
The result is typically an error from the HS trace such as: Failed to load ODBC library symbol: /u01/app/TimesTen/tt1122/lib/libtten.so(SQLAllocHandle)+
DataDirect Unix/Linux driver manager for Oracle TimesTen to the rescue
DataDirect is the only commercial ODBC Driver Manager recommended by Oracle per Doc ID 813942.1, and our Unix/Linux driver manager is able to support the Oracle TimesTen client from the Oracle Database Gateway, as well as other downstream applications including other real-time database federation technologies like Sybase ECDA. The driver manager can also load the Oracle TimesTen client directly from other non-Oracle business applications including SAS/Access, IBM Cognos, or SAP Business Objects.
Image may be NSFW. Clik here to view.
Oracle TimesTen Gateway
Setup the Gateway to Oracle TimesTen
1. Assume Database Gateway for ODBC (DG4ODBC) is installed (article is for Linux x64). DG4ODBC is included with Oracle Databases, or available for download from Oracle.com if you want to set it up on a separate tier.
2. Download the Connect for ODBC Unix/Linux Driver Manager which is included in the standard DataDirect Connect64 for ODBC package and comes included with any ODBC driver installation.
4. The specific steps include adding the DataDirect Unix/Linux driver manager path and file “libodbc.so” (or odbc.so on AIX) to the HS_FDS_SHAREABLE_NAME initialization property configured in the . The location of the shared library path to the driver manager library should also be added to the ENVS variable in the $ORACLE_HOME/network/admin/listener.ora file.
5. Below are example data sources to add to the odbc.ini file:
Use tnsping CACHEDB1_CS to check whether there is any issue or not. It should be ok.
9) Create and test the database link as follows:
SQL> CREATE DATABASE LINK TT CONNECT TO :”user” IDENTIFIED BY “password” USING ‘CACHEDB1_CS’;
SQL> select * from ttuser.test@TT;
A
———-
1
2
3
Got data connectivity questions?
Call 1-800-876-3101 to speak with a Systems Engineer to learn how other organizations are making progress with real-time gateway connectivity to Oracle TimesTen, as well as other data sources for which we supply both the driver manager and ODBC driver including SQL Server, DB2, Salesforce, Hadoop Hive, MongoDB, Cloudera Impala and more.
I am hearing more about Alteryx as a means for organizations to supplement a big data Hadoop platform with key dimensions from business applications. With the migration of several business operations to SaaS applications, DataDirect Cloud is providing unlimited potential to Alteryx data objects for analysis.
DataDirect Cloud supports the ODBC standard that Alteryx uses for input data to sources for instant access by data artisans. The growing list of available connectors include those to the latest APIs for Salesforce, Eloqua Marketo, Hubspot, Force.com, Rollbase, as well as connecting ODBC over HTTPS to RDBMS systems such as SQL Server, Oracle, or DB2 that reside in a public cloud or behind a firewall. These ODBC SaaS connectors expose data in real-time through an ANSI SQL interface, similar to SQL Server or Oracle, so you can import data in a standard way from Alteryx via tables, Visual Query Builder or SQL Editor.
Getting started with Alteryx and its new SaaS connectors from DataDirect Cloud
2. From DataDirect Cloud, select a data source. In this case, I am selecting Eloqua since I have a production account.
3. From DataDirectCloud, configure your connection information to Eloqua. My Eloqua admin setup an account and I received an e-mail from support@eloqua.com with the connection information.
4. Download the DataDirect Cloud ODBC client to the Alteryx system and configure the local ODBC data source to connect to Eloqua.
5. Launch Alteryx Designer
6. Select In/Out tab and drag “Input Data” icon to the designer pane
7. Select “Input Data” and create “New ODBC Connection”
Image may be NSFW. Clik here to view.
8. Select the System ODBC data source we created for Eloqua using the DataDirect Cloud ODBC client available in both 32-bit and 64-bit versions. Use your Progress ID to authenticate with DataDirect Cloud.
Image may be NSFW. Clik here to view.
9. Select the table object to import data, construct a query, or write a SQL statement similar to any relational data source. It’s pretty cool when you execute SQL statements against SaaS applications complete with joins, aggregation, and scalar functions.
13. Now that Alteryx is connected to new and exciting data sources, I will defer to the data artisans to do really cool stuff.
Get started loading unlimited data into Alteryx
Start a trial of the DataDirect Cloud Connectivity Service to load data from any ODBC source into Alteryx or call 1-800-876-3101 to speak with a systems engineer on ODBC data connectivity to SaaS, Big Data, SQL, NoSQL, and more.
Following Data Summit 2014 in Manhattan, I spoke with panels of industry experts about NoSQL technologies related to financial services. Coincidentally, MongoDB headquarters is down the street and they have very clever business cards.
Image may be NSFW. Clik here to view.
I was able to speak with conference attendees about their use of MongoDB and learned how valuable it is for ingesting and storing raw data for operational use including tick data, FIX messages, SWIFT, FpML, etc. I learned about several high data volume use cases being driven by regulations such as Dodd-Frank and EMIR. The consensus was that standards based SQL access provides greater potential for adoption of MongoDB by seamlessly integrating from a wide range of BI or warehousing tools for regulatory reporting and analytics directly from the source system.
Avoid the loss of data from flattening collections for SQL
The feedback I hear from financial services shops with SQL access to NoSQL is that the sampling of data to define a schema introduces potential for data loss, which is not an option for regulatory reporting. What happens to embedded arrays? What if the “schema” on read changes? Will my ETL, Pig or Map/Reduce jobs break as document structures change?
A new SQL approach to MongoDB so you can have your cake and eat it too
DataDirect introduces the first reliable MongoDB ODBC connector that provides flexible normalized “schema on read” functionality to query MongoDB using SQL, while optimizing SQL queries against the expanding MongoDB query API. This logical schema does not require additional ETL steps or data movement, and is yet another engineering marvel for DataDirect R&D.
The case for standards based connectivity to MongoDB
1. BI Infrastructure: Instantly connect any versions of your BI tools across SAP Business Objects, IBM Cognos, Tableau, Oracle Business Intelligence (OBIEE), Microstrategy, SQL Server Reporting Services, SAS, SPSS, Qlikview, Spotfire, and more. Or extract the data with less development time using ETL platforms such as IBM DataStage, Informatica, Ab Initio, SSIS and more. A JDBC driver will be available as well for those building organic warehousing infrastructure.
2. Database Federation: Given the tiered economics of database storage, MongoDB can serve as a lower cost staging area for either structured or unstructured data that is accessible directly in real-time using ODBC in conjunction with RDBMS federation technologies including Oracle Database Gateway and SQL Server Linked Server.
Image may be NSFW. Clik here to view.
3. Democratization of MongoDB: Standard SQL access to data enables a larger audience to derive business value. Especially for self service analysts or citizen developers who are unfamiliar with MongoDB APIs. It’s common for them to run Microsoft Office, Data Visualization Tools, or even SaaS BI applications that have open JDBC support such as Birst.
Are we the Splice of MongoDB?
We’re not guaranteeing ACID compliance, but it sure does look like an RDBMS through our connection.
If you’re a business intelligence or data warehousing professional in financial services for which MongoDB is one of many sources you consume, here’s a great webinar to learn more about why your operations team adopted MongoDB: How to Drive Business Value in Financial Services with MongoDB
Things are off to a great start at MongoDB World, and the show couldn’t have come at a better time, as we’ve just announced the expansion of Progress DataDirect on-premise connectivity to support MongoDB and Amazon Redshift. DataDirect now offers data integration for MongoDB and Amazon Redshift via both ODBC and JDBC standards.
Image may be NSFW. Clik here to view.
Many of today’s business critical systems are being developed with MongoDB and Amazon Redshift, so the demand to connect to such data is growing rapidly. Progress DataDirect provides simple, scalable, secure real-time connectivity to these sources and others without the traditional complications of doing so, such as data loss and data “flattening.” Now users can easily connect MongoDB data with their existing data management, analytics or enterprise data warehouse (EDW) ecosystems.
In addition to the expanded connectivity, we’ve also added the following list of certification updates:
SQL Server 2014
Teradata 14.10
DB2 V11 for z/OS
Windows 8.1
If you’re at the conference, make sure you come by our booth #119 to learn more about DataDirect and see firsthand the new connectivity to MongoDB. Or, if you want to learn more about Progress’ newest acquisition, Modulus, stop by their booth right next door! Modulus is a cloud platform that allows developers to build new applications using Node.js to connect to MongoDB data.
Tableau analysts may or may not know about the challenges in exposing NoSQL data sources, such as MongoDB, to data visualization tools like Tableau, Qlikview, Spotfire, Microstrategy, SAP Lumira, SAS, Birst, etc. Progress DataDirect loves a good challenge and engineered the first reliable SQL connector to MongoDB using ODBC/JDBC. Progress Software acquired the Modulus Node.js and MongoDB development platform, so we might know a thing or two about MongoDB data – rather than follow the widely unpopular flattened schema approach.
Getting Started on reliable Tableau connectivity to MongoDB:
1- Download and install theDataDirect MongoDB ODBC driver (32-bit or 64-bit depending on version of Tableau). Tutorial shows Tableau 8.2 working with the 64-bit MongoDB ODBC driver.
2- Create an ODBC Data Source to connect to MongoDB (see end of tutorial for sample data with arrays of embedded documents)
Image may be NSFW. Clik here to view.
3- Launch Schema Tool and select “Normalized View” to create an intelligent virtual schema with a single click of the OK button
7. For comparison, it’s impossible to ask the same question as step #6 when the data is exposed in a flattened schema like existing SQL connectors. Note how each individual team’s players and goals are all separate measures and dimensions from the flattened view of the Worldcup data (TEAMS1_GOALS, TEAMS2_GOALS, TEAMS3_GOALS, etc). This approach does not represent MongoDB or NoSQL technology very well from Tableau.
Image may be NSFW. Clik here to view.
What’s the alternative to DataDirect’s normalized and scalable virtual schema for Tableau?
To get an equivalent reliable connectivity experience from Tableau, you would need to:
Sample MongoDB data to understand the data model
Write mapping code to extract and normalize the data
Physically load it into a relational database
Create some kind of CDC capability for near real-time data connectivity
In summary, use the DataDirect MongoDB ODBC driver.
Many powerful vertical solutions are now available on the Force.com platform via the AppExchange – For ITSM (IT Service Management), there is BMC’s Remedyforce. Navatar Group offers Financial Services solutions, and ERP and accounting solutions are available from FinancialForce and RootStock. Veeva offers a CRM solution for the life sciences, to highlight just a few.
Force.com solutions provide varying capabilities when it comes to reporting or analytics. Being based on the Force.com platform, the integrated reporting or analytics will likely be delivered via your browser. If you’re a Force.com developer, this means that you will be providing some level of reporting or analytics in your application. If you’re a user of an application built on the platform, you will likely be using the BI or reporting features delivered with your solution. (I’ll be using a Remedyforce sandbox here.)
What if you’d like to access your data from other applications? Maybe you’re more comfortable with SQL than SOQL? If you’re a developer, are requests for custom reports from your customers driving you crazy? If you’re a user of a Force.com solution, perhaps your company has standardized on particular BI or analytics applications – what if you could use them with your Force.com data ? But of course, all of this can only be done if it can be done securely.
Developers or end-users – unlock your data! You can now access the data that’s in virtually any Force.com solution that stores data in custom or standard Salesforce objects from your favorite BI/reporting/analytics applications! App Dev environments. Even from data federation applications such as Oracle’s Database Gateway for ODBC!
For example, with Datadirect Cloud, Remedyforce users, as well as users of almost any Force.com solution that stores data in standard or custom objects, can use ODBC-compatible visualization applications like Tableau to access their data.
Remedyforce integration with Adobe ColdFusion Builder via JDBC and Datadirect Cloud
Datadirect Cloud gives you realtime SQL access via ODBC or JDBC to your data stored in standard or custom Salesforce objects. Powerful, easy-to-setup and easy-to-use. No ETL required! And fully secured over SSL and HTTPS.
Do you work with Force.com solutions? Enduser or developer – unlock your data TODAY. Try Datadirect Cloud for free.
From working on data connectivity projects for a decade, I can say with certainty that hybrid cloud / on-premise environments have become the norm. And I’m hearing requirements for secure data connectivity to data stores that reside on-premise behind a fire wall; or in a public cloud server running on Amazon. Rather than open a database port, VPN, or SSH tunneling; you can expose that data using the DataDirect Cloud On-Premise Connector with real-time and direct ODBC/JDBC data connectivity. Today, we introduced support for OpenEdge 10.2B to 11.3, and Oracle, SQL Server and DB2 are coming soon using the same proven approach.
User Tested, CIO Approved
The DataDirect Cloud On-Premise connector capability certainly raises questions for the on-premise world, especially in very large enterprises. Last week, I demo’d the on-premise connector technology to Progress Software’s CIO, Dave Benson, and his team. After addressing their security questions, the team saw the benefits with some current initiatives at Progress Software, and approved of the architecture. Progress IT recommends partnering IT with business sponsors for collaboration on data connectivity projects in case there are concerns around access to an IT managed database.
Prezi for ODBC/JDBC from SaaS/PaaS over HTTPS to on-premise OpenEdge, Oracle, SQL Server, DB2
Highlighting capabilities of DataDirect Cloud and Rollbase
The prezi for my Progress Communities Webinar is followed by two live demos. First, I show the hosted version of Rollbase federating external objects directly from on-premise RDBMS, the ubiquitous OpenEdge sports database. Then, we can built a chart of items in the hosted SaaS BI application, Explore Analytics. Finally, to prove the direct real-time connectivity to on-premise; we update a federated record from Rollbase and re-run the charts in Explore Analytics to ensure the updated record is coming back. At this point, attendees will be mesmerized by the data connectivity!
Infosphere DataStage is one of the top data integration platforms from a data connectivity perspective since most sources and targets are fueled by DataDirect ODBC connectivity. This includes our streaming bulk load (without landing flat files) functionality for faster loads into popular RDBMS such as SQL Server from Unix/Linux.
I have been working with an increasing number of DataStage shops looking for expanded connectivity to SaaS applications using flexible ODBC access direct to Salesforce, Veeva CRM, Eloqua, Oracle RightNow and Marketo. DataDirect delivers SaaS data connectivity that is built from the same framework as relational ODBC drivers already bundled with DataStage today.
Why add DataDirect SaaS connectivity to your stage?
We’re putting a SQL-92 ODBC compliant interface on SaaS applications so ETL developers can hit the ground running. You don’t even have to know how to spell “eloqua”.
Disruptive SaaS data sources are coming soon to a DataStage near you. I’ve seen a couple projects where organizations are forced to use a kludgy architecture to consume SaaS data against your own best practices/data governance for consuming SaaS data.
IT can take control of the SaaS landscape before business sponsors even know what to ask. It’s really important to me that you use the best platform for the job without spinning up one-off satellite data integration systems.
The same connectivity works with any standards based IBM application including IBM Cognos (Data Manager, BI, Transformer), IBM Cognos TM1, IBM SPSS, DB2 Federation Server, Websphere Application Server … as well as any standards based applications from Oracle, SAP, or Microsoft.
Getting Started unleashing the power of DataStage on SaaS
Below are steps to integrate additional DataDirect ODBC drivers with DataStage:
Verify test-connection works independently from DataStage by following the corresponding quick start guide with the download.
Integrate the ODBC shared library path (e.g. LIBPATH on AIX, SHLIB_PATH on HP-UX pa-risc, and LD_LIBRARY_PATH on Solaris, Linux and HP-UX Itanium) in <InformationServer>/DSEngine/dsenv file. (Note: If running on Websphere, the shared library path will need to be configured from Websphere)
Specify the ODBC data source name in the uvodbc.config
Merge the odbc.ini data source into the existing file used by DataStage typically found in $DSHOME. Below is a sample Salesforce DSN:
[Salesforce]
Driver=/opt/Progress/DataDirect/Connect64_for_ODBC_71/lib/ddsfrc27.so
Description=DataDirect 7.1 Salesforce
HostName=login.salesforce.com
SecurityToken=<your security token>
StmtCallLimit=0
TransactionMode=1
Database=/tmp/sfdc
Optional: The Connect XE Salesforce.com driver requires a JRE, and if DataStage is running with an older JRE version, the driver can be configured to use a standalone JRE independent from DataStage by following these steps:
– Configure shared library path with desired JRE. For example on AIX: – LIBPATH=/usr/java6_64/jre/lib/ppc64:/usr/java6_64/jre/lib/ppc64/classic:$LIBPATH
– Add following property to Salesforce ODBC DSN in odbc.ini file: SQLEngineMode=1
– Check whether the standalone java service is running, ps -ef | grep “sforce”
– If not running, please start it from the following directory: /opt/Progress/DataDirect/Connect64_for_ODBC_71/java/lib
– java -cp sforce.jar com.ddtek.cloud.sql.Server &
Restart the node agents and begin importing SaaS metadata via ODBC
Image may be NSFW. Clik here to view.
Information Server Designer Client ODBC connection to SaaS
Learn more about the expanded Data Connectivity Options available from DataDirect
2. Or call 1-800-876-3101 to speak with a systems engineer to learn how other organizations are making progress with DataDirect Connect Salesforce integration.
A few weeks back a colleague and I were driving through Seattle, and we began to debate the reach of ODBC as a key data standard. His perspective was that ODBC was really a C/C++ technology, and that as people have largely moved to other programming languages, the importance of ODBC has declined as well. Well, I didn’t buy that argument but needed to do some research before I could refute him. As I mainly tinker with iOS development these days, my awareness of what is happening in the programming world at large is well out-of-date.
Well, it didn’t take long to dig up some examples:
ODBC for PHP. Basically, as long as you have an ODBC driver, PHP will support it just lovely. Here is the manual on how to configure it: http://us2.php.net/manual/en/book.uodbc.php
ODBC for Erlang. Well, that is pretty cool. Trying to write a highly parallelized application that deals with a lot of data? Can’t get much better of a combo than this! Here’s the manual pages that show how to set this up too: http://www.erlang.org/doc/man/odbc.html
ODBC for Python. Okay, I think I’m starting to prove my point. Here’s the link to setting this up: https://wiki.python.org/moin/ODBC
What is interesting is that most of the examples here refer to relational databases. But in case you didn’t already know, what Progress DataDirect also does really well is make any cloud application and Big Data, NoSQL database look and feel just like a relational database. So let’s say you want to do things like:
For “big data” connectivity, I see a very high percentage of projects delivering business value from big data initiatives that require data connectivity from DataDirect. This was specifically around providing Universal data connectivity to Hadoop Hive and related low latency interfaces (Impala, Hawq, Drill, Stinger, etc).
From recent active projects, we’re connecting systems in the other direction which takes success to new levels; and that is when you can move data sets aggregated by the Hadoop big data ecosystem; and move them to relational databases managed by strict corporate security policies which means your data is above the radar now. The most common is the requirement to connect to SQL Server from Linux using Windows Authentication (example from Hortonworks community).
How ODBC and JDBC SQL Server Windows Authentication from Linux works:
DataDirect is your exclusive partner in data connectivity providing unique support for Windows Authentication from Linux across both ODBC and JDBC connectivity. Sqoop is a JDBC utility to move data between Hadoop Ecosystems and relational databases such as SQL Server. On the ODBC side, we’re seeing Python/Ruby wrappers that need to connect to relational DBs. Microsoft recommends using integrated security over mixed mode authentication for secure SQL Server platforms.
DataDirect invented “wire protocol” technology to connect to data without data source specific dependencies; and our implementation for windows authentication from Unix/Linux is no exception. You simply need to configure the driver with your Windows credentials in plain text. For Kerberos support, no credentials would be required.
Getting started with ODBC and JDBC SQL Server NTLM authentication from Linux:
Not even the sexiest data scientist and Hadoop platform are above corporate security, so here’s how you can connect from JDBC and ODBC to SQL Server to remain in compliance:
[SQLServer]
Driver=/opt/Progress/DataDirect/Connect64_for_ODBC_71/lib/ddsqls27.so
Description=DataDirect 7.1 SQL Server Wire Protocol
AuthenticationMethod=9
Database=test
Domain=AMERICAS
EnableBulkLoad=0
#Connecting to SQL Server Named Instance
HostName=172.30.57.75\SQL2008R2SP2
LoadBalancing=0
Pooling=0
# PortNumber not required when connecting to Named Instance
#PortNumber=1433
During test connect with the DataDirect samples/example utility, specify the following credentials for windows ntlm authenticated user “AMERICAS\SUMITS” and note domain is not required as part of user name since it was configured in the data source:
sumits@nc-lnx16 /opt/Progress/DataDirect/Connect64_for_ODBC_71/samples/example> example
example DataDirect Technologies, Inc. ODBC Example Application.
Enter the data source name : topptt01_cs
Enter the user name : SUMITS
Enter the password : my_windows_auth_pwd
Enter SQL statements (Press ENTER to QUIT)
SQL>
Download the DataDirect Connect for JDBC drivers for and follow the quick start guides supplied with the download. The SQL Server JDBC connection URL should be configured as follows for NTLM authentication:
