chacadwa.com

Technical blog and writings by Micah Webner.

Building a Curriculum Management and Publication System using ERP Data and Web Services

DrupalCamp Michigan 2013
Micah Webner
Web Architect
Henry Ford College

About me

  • Contract IT Services employee at Henry Ford College
  • 24 years in IT, 16 at HFC
  • Building websites since 1995, as primary job function for 4 years
  • drupal.org user for just under 8 years
  • Started on Drupal 4.7.x

Introduction

Henry Ford College's Drupal-based curriculum management system utilizes data extracted from the ERP system (aka HANK) as custom entities to promote data standards, simplify user workflows and reduce data entry redundancies. Drupal content on the curriculum management website is then shared with HFC's public-facing websites using the Services module.

What this session will cover

I could give a detailed overview of this web project in about 10 minutes. I could easily talk about it for four hours, maybe even double that.

I have 40 minutes.

So we are going to go really fast and highlight all of the things I had to get working together to make this project happen. Along the way, I'll include code examples and links to the blog posts that I read to figure this stuff out.

Drupal's APIs really are powerful enough to make most of this happen with minimal work on the developer's part.

The real hard part is figuring out what pieces and steps are needed. That's what I hope you will take away from this session.

All of my repos for this project are published on Bitbucket.org, so you're free to examine my code to see all of the ugly hacks I used to make everything work.

Three main topics:

  1. Custom Entities: Externally-populated data tables configured as fieldable Drupal entities.
  2. Web Services Provider: Drupal content easily exposed using the Services and Services Views modules.
  3. Web Services Client: Custom clients that can retrieve and render this information on other sites.

What this session will not cover

  • Curriculum Management: Sorry, I picked the title before I wrote the session. Most of the actual "curriculum management" portions of this site really just come down to custom content types and the Workbench Moderation module. There are plenty of DrupalCon sessions and tutorials online explaining how all of that works.
  • Security: This system does not use any authentication security for web services. Our back-end web servers are protected by NetIQ Access Manager, and all (well, most) of the content exposed by this system is for public audiences anyway.
  • Every detail: A lot of the coding required for this system is somewhere near the high end of Intermediate, and won't fit in a 40-minute session. What I hope to cover is a good overview of the elements that need to be built, with links to enough tutorials and examples that you can see what's possibile and then muddle through something similar without making a lot of the same mistakes I have.
  • External data sources. Suffice it to say that our enterprise reporting tool can read Oracle data and write output as MySQL tables.

What's wrong with this system

  • Weak coding of the HTTP client. This system uses drupal_get_http() to retrieve data. The system would be more robust and secure if it used the HTTP Client module Guzzle. As it is, this code is running in production and is quite stable, probably because all the websites are on the same physical server.
  • Awkward handling for file attachments. Attached files on the back-end system are hard to address. We have to either link to files on the central server or pass their contents using Services. Ideally, these should move to a CDN.
  • Inconsistent approach. My approach is constantly changing as this system grows. I rarely have time to go back and update older parts of it with newer techniques.

Prerequisites for this session

Some people might consider this to be an advanced session, but I say it's still intermediate. Advanced to me is digging into the guts of Core's database abstraction layer, implementing Symfony, or messing with Views plugins.

To build a system like this you'll need:

  • Module-building basics. How to create a .info file and start writing modules.
  • Understanding of Drupal's hook system and typical APIs like hook_menu().
  • How to write page callbacks, and maybe blocks.
  • A basic understanding of Drupal Render Arrays and how they work.

Even if you don't know that stuff, I hope you come away from this session understanding the possibilities discussed here, and some of the pieces needed to make it happen.

A quick project oveview

Problems we're solving

  • Duplicated content
    • Online learning and divisions both wanted to post course information.
    • Published information out of synch with catalog.
  • Bad documentation workflow
    • Course Masters tracked in (1000+) Word and WordPerfect documents.
    • Course information went from Word to ERP, Website and print/PDF catalogs via different means, including handwritten markup and sticky notes.
  • Lack of data standards
    • Web systems used different division/department codes than ERP system.
    • External systems had field sizes bigger than ERP system.
    • Legacy web Program listings had five fields for each of two contacts on each program.

Types of data involved

  • From ERP
    • Divisions
    • Departments
    • Academic Terms
    • Buildings
    • Subjects
    • Transfer-In Course Equivalencies
    • Staff directory
  • From Courses Website
    • Course Masters
    • Catalog Course Entries
    • Degree and Certificate Programs
    • Office Contact / Hours of Operation
    • Curriculum approval forms
  • From HFC Newsroom - BANE
    • Bulletins (page banner alerts)
    • Announcements (stories about future things)
    • News (stories about things that have happened)
    • Events (calendar items)

For this session, we will focus on the Staff Directory, which we've implemented as a Drupal fieldable entity. This data is used by every part of the system, so it can be a useful example for almost everything we're going to cover today.

Part 1. Creating custom entities

This section will cover the basics of defining custom entities and discuss what is required to make them fieldable, editable and ready for display alone, with Views or as options in select lists.

What are entities?

  • Primary method of data storage in Drupal 7.
  • In core: Nodes (content), Users, Taxonomy terms, Comments
  • In contrib: Drupal Commerce Products, Entity Forms.
  • Any data stored in one (or more) tables with a unique key that you want to perform CRUD functions on.

Why custom entities instead of nodes?

  • Data that needs hard-coded required fields for calculations, etc, in custom code.
  • Simple data that doesn't need full CRUD.
  • Externally-generated data.

There are a bunch of things you need to learn in order to manipulate custom entities. The hardest part seems to be finding a single tutorial that covers them all. (This probably isn't it, either.) But once you know what you need to do, then it's just a matter of making lots and lots of arrays that define almost the same thing, just in very slightly different ways. Given the lack of time to cover this in detail, this session is going to skim over the highlights of what you'll need to know about.

See also:

Entity API Module

  • Provides a lot of missing functionality that isn't part of Drupal 7 Core, mainly for CRUD and Views.
  • If you're doing any work with custom entities, just use it.

See also:

hook_schema()

So here's our table in phpMyAdmin:

example table in phpMyAdmin

In Drupal 6 (and earlier?) if you wanted to access external or custom data, you could define it using hook_schema() in a module's .install file and then manipulate it with database queries and drupal_write_record(), etc. Drupal 7 handles this schema information the same way.

The easiest way to make this is to create your database table(s), then use the Schema Module to copy and paste the definitions.

Schema compare

The new table(s) will appear in the Extra tables section on the compare tab.

Schema Compare

Schema inspect

Switch to the Inspect tab and copy the definition you want to add.

Schema Inspect

Add to .install file

Here's the hook_schema() call from my staff directory module.

/** * Implements hook_schema(). */ function staffdir_schema() { $schema = array(); $schema['hank_hrper'] = array( 'description' => 'HFC Staff Directory WebFOCUS Report', 'fields' => array( 'hrper_id' => array( 'description' => 'Employee HANK ID', 'type' => 'int', 'not null' => TRUE, ), 'username' => array( 'description' => 'HFC Username', 'type' => 'varchar', 'length' => '30', 'not null' => FALSE, ), 'first_name' => array( 'description' => 'First Name', 'type' => 'varchar', 'length' => '30', 'not null' => FALSE, ), 'last_name' => array( 'description' => 'Last Name', 'type' => 'varchar', 'length' => '57', 'not null' => FALSE, ), 'email_address' => array( 'description' => 'Email Address', 'type' => 'varchar', 'length' => '50', 'not null' => FALSE, ), 'primary_pos_id' => array( 'description' => 'Primary Position ID', 'type' => 'char', 'length' => '14', 'not null' => FALSE, ), 'pos_title' => array( 'description' => 'Position Title', 'type' => 'char', 'length' => '60', 'not null' => FALSE, ), 'pos_division' => array( 'description' => 'Division Code', 'type' => 'char', 'length' => '7', 'not null' => FALSE, ), 'pos_dept' => array( 'description' => 'Department', 'type' => 'char', 'length' => '10', 'not null' => FALSE, ), ), 'primary key' => array('hrper_id'), ); return $schema; }

TODO: Add some more indexes.

See also:

hook_entity_info()

Now that Drupal understands our table, we need to define it as an entity with hook_entity_info().

/** * Implements hook_entity_info(). * * @see http://www.istos.it/blog/drupal-entities/drupal-entities-part-3-programming-hello-drupal-entity * @see http://drupal.org/node/1026420 */ function staffdir_entity_info() { return array( 'staffdir' => array( 'label' => t('Staff directory'), 'entity class' => 'StaffDir', 'controller class' => 'StaffDirController', 'views controller class' => 'EntityDefaultViewsController', 'base table' => 'hank_hrper', 'label callback' => 'entity_class_label', 'uri callback' => 'entity_class_uri', 'access callback' => 'staffdir_access', 'fieldable' => TRUE, 'entity keys' => array( 'id' => 'hrper_id', ), 'static cache' => TRUE, 'bundles' => array( 'staffdir' => array( 'label' => 'Staff directory', 'admin' => array( 'path' => 'admin/structure/staffdir', 'access arguments' => array('administer staffdir'), ), ), ), 'view modes' => array( 'full' => array( 'label' => t('Staff directory'), 'custom settings' => FALSE, ), ), ), ); }

There are a few noteworthy entries here:

  • The defaults for entity class, controller class and views controller class provided by the Entity API module are Entity, EntityAPIController and EntityDefaultViewsController respectively. We're going to extend Entity and EntityAPIController slightly for our purposes.
  • base_table matches our database table name, and its hook_schema() entry.
  • label callback and uri callback functions are provided by EntityAPI. We will manipulate those when we extend the Entity class.
  • This entity is fieldable. Drupal 7's field system requires that entity keys are integers. If we have time, I'll show how to add custom field values to tables with string keys.
  • We've defined a view mode of full. Think of full or teaser modes in nodes. Generally speaking, we should define at least one view mode for each entity type, but I sometimes skip building actual displays for entities that don't need them. (Example: buildings or course subjects don't need full displays.)
  • We've define a single bundle whose name matches our entity name. Bundles are like sub-types of entity types. (Think node types, Commerce product types, different entityforms variations.)
  • The admin path information is magic. Set this path, add it to hook_menu() with any page callback, and you automatically get the Manage Fields and Manage Display tabs attached to it.
  • The access callback is required by entity_access().

admin path

Here are the Fields tabs attached to the administration form.

Field management on entity admin page

See also:

entity_access()

Your access callback needs to specify access levels for view, create, update and delete operations.

/** * Access callback for staffdir. * * Arguments for this function are defined by entity_access(). * @see entity_access() * */ function staffdir_access($op, $entity = NULL, $account = NULL, $entity_type = NULL) { switch ($op) { case 'view': return user_access('access content'); case 'create': case 'update': case 'delete': return user_access('edit any staffdir'); } }

I'm generally in a hurry when I write this part, so I tend to overuse the administer content permission here.

hook_menu() auto-loader wildcards.

Most tutorials on creating entities have a lot of confusing stuff about creating entity_type_load() and ***entity_type*_load_multiple()** functions, and if you name these wrong, you'll blow up your module and spend hours figuring out where you went wrong.

To some degree, these functions are only really necessary when you're not using Entity API, which provides an entity_load() function that can be called directly without needing these other functions at all.

What you really need to understand is Auto-Loader Wildcards in hook_menu()

Given the following entries in hook_menu():

$items['staffdir/%staffdir'] = array( 'title callback' => 'entity_label', 'title arguments' => array('staffdir', 1), 'page callback' => 'staffdir_page_view', 'page arguments' => array(1), 'access callback' => 'entity_access', 'access arguments' => array('view', 'staffdir', 1), 'type' => MENU_CALLBACK, 'file' => 'staffdir.pages.inc', ); $items['staffdir/%staffdir/view'] = array( 'title' => 'View', 'type' => MENU_DEFAULT_LOCAL_TASK, 'weight' => -10, ); $items['staffdir/%staffdir/edit'] = array( 'title' => 'Edit', 'page callback' => 'drupal_get_form', 'page arguments' => array('staffdir_form', 1), 'access callback' => 'entity_access', 'access arguments' => array('update', 'staffdir', 1), 'type' => MENU_LOCAL_TASK, 'file' => 'staffdir.pages.inc', );

The %staffdir wildcard has nothing to do with the module name or the entity type. It is simply a prefix that expects a function named staffdir_load() to provide the value that will be passed as a page argument.

To make this clearer, let's say we've defined our menu item as $items['staffdir/%staffdir_entry'] instead. Now Drupal would expect to find a function called staffdir_entry_load() instead.

Note: I don't understand this, but if your function wildcard matches your entity name, but not your module name, then weird things happen because your loader function will also get picked up by module_invoke() during the entity_load() too. So just don't name it that. (I can't explain it better than that, but if you get a bunch of weird errors about array_flip(), try renaming the wildcard and loader function.)

At a minimum, then, we'll want our callback function to call entity_load() for our actual $items['staffdir/%staffdir'] and $items['staffdir/%staffdir/edit'] entries.

/** * Load a single record. * * @param $id * The id representing the record we want to load. */ function staffdir_load($staffdir_id) { $entity = entity_load('staffdir', array($staffdir_id)); // If we got results, we only want the first element of the array. return $entity ? reset($entity) : FALSE; }

See also:

hook_field_extra_fields()

hook_field_extra_fields() exposes your entity's fields to the Manage fields and Manage display tabs. As far as I can tell, this is something you probably should do, but I'm not convinced it really works, so I'm not going to detail it here, and I don't generally bother with it.

hook_entity_property_info_alter()

You will want to implement hook_entity_property_info_alter(), especially if you want to use your custom entity with Views. Without this code, EntityDefaultViewsController will still expose your table fields to views, but without any formatting or descriptive text, and labels will just be field names.

/** * Implements hook_entity_property_info_alter(). */ function staffdir_entity_property_info_alter(&$info) { $properties = &$info['staffdir']['properties']; $properties['hrper_id'] = array( 'label' => t("HANK ID"), 'type' => 'text', 'schema field' => 'hrper_id', 'description' => t("The person's HANK ID."), ); $properties['username'] = array( 'label' => t('HFC Username'), 'type' => 'text', 'schema field' => 'h19_idv_oee_username', 'description' => t("The person's HFC network login name."), ); $properties['first_name'] = array( 'label' => t("First Name"), 'type' => 'text', 'schema field' => 'first_name', 'description' => t("The person's first name."), ); // Skip some fields to save space. $properties['employ_date'] = array( 'label' => t("Employment date"), 'type' => 'date', 'description' => t("The date the person was hired."), 'schema field' => 'employ_date', ); }

Extending entity and controller classes

Extending the Entity class

As mentioned above, we're going to want to extend the Entity class to provide custom responses for the label callback and uri callback we specified in hook_entity_info() earlier. To do this, we will override defaultLabel() and defaultUri() methods.

/** * StaffDir class extends Entity. */ class StaffDir extends Entity { protected function defaultLabel() { return check_plain($this->first_name . ' ' . $this->last_name); } protected function defaultUri() { return array('path' => 'staffdir/' . $this->identifier()); } }

The new defaultLabel method concatenates our first_name and last_name fields for a nicer entity title. The new defaultUri() matches the pattern we created in hook_menu() above.

Extending the EntityAPIController class

We will want to extend the EntityAPIController class if we want to affect any of the default CRUD functions it provides. For example, let's say we've built our forms to edit the custom fields we've attached. If we now try to display our entity with entity_view(), only the custom fields will be shown. We need to manipulate the buildContent() method to add additional elements to the $content render array.

/** * StaffDirController extends EntityAPIController. * * Our subclass of EntityAPIController lets us add a few * customizations to create, update, and delete methods. */ class StaffDirController extends EntityAPIController { public function buildContent($entity, $view_mode = 'full', $langcode = NULL, $content = array()) { // Only display Employee ID to editors. if (entity_access('edit', 'staffdir', $entity)) { $content['hrper_id'] = array( '#theme' => 'item', '#weight' => -10, '#label' => t('HANK ID'), '#markup' => check_plain($entity->hrper_id), ); } $content['primary_pos_id'] = array( '#theme' => 'item', '#weight' => -8, '#label' => t('Primary Position ID'), '#markup' => check_plain($entity->perstat_primary_pos_id), ); $content['pos_title'] = array( '#theme' => 'item', '#weight' => -7, '#label' => t('Position title'), '#markup' => check_plain($entity->pos_title), ); return parent::buildContent($entity, $view_mode, $langcode, $content); } }

Creating edit forms for fields.

For fieldable entities, you'll need to build Drupal forms to allow editing those fields.

Staff Directory edit form

There are four functions needed for this:

In this Staff Directory example, it doesn't hurt us to use entity_save() to write the changes, since we're not changing the base table and it is overwritten every night, anyway. If your situation makes this unsafe, you'll probably have to work out the code to call either field_attach_insert() or field_attach_update() instead.

The edit form /** * Display the entity edit form. * * We can't edit the actual entity items, but this allows us to manage * the contents of attached fields. */ function staffdir_form($form, &$form_state, $entity) { $uri = entity_uri('staffdir', $entity); drupal_set_title(entity_label('staffdir', $entity)); drupal_set_breadcrumb(array( l(t('Home'), '<front>'), l(t('Staff Directory'), 'staffdir'), l(entity_label('staffdir', $entity), $uri['path']) )); $form = array(); // This exists for now because entity_example.module says so. // ...and we appear to need it later in _submit... $form['entity'] = array( '#type' => 'value', '#value' => $entity, ); // Do whatever you want here to display the uneditable fields. field_attach_form('staffdir', $entity, $form, $form_state); $form['submit'] = array( '#type' => 'submit', '#value' => t('Save'), '#weight' => 100, ); return $form; } Validation /** * Validate the entity edit form. * * We don't have any native fields to validate, but we want to * allow Field API to do any validation that it requires. */ function staffdir_form_validate($form, &$form_state) { $values = (object) $form_state['values']; field_attach_form_validate('staffdir', $values, $form, $form_state); } Form submission /** * Entity editing form submit. */ function staffdir_form_submit($form, &$form_state) { $entity = $form_state['values']['entity']; field_attach_submit('staffdir', $entity, $form, $form_state); entity_save('staffdir', $entity); $form_state['redirect'] = 'staffdir/' . $entity->identifier(); }

Providing default values

When you're not using external data, it can be handy to override the entity_create() behavior to provide some default values to your entity. Here's a simple example of replacing the create() method:

/** * Create and return a new widget entity. * * Overrides EntityAPIController::create(). */ public function create(array $values = array()) { global $user; $values += array( 'widget_id' => 0, 'uid' => $user->uid, 'created' => REQUEST_TIME, 'changed' => REQUEST_TIME, ); return parent::create($values); }

Custom fields the hard way

In order to make entities fieldable, you must have integer key ids. The divisions and departments tables from our ERP system have varchar keys.

$schema['hank_divisions'] = array( 'description' => 'HFC Divisions', 'fields' => array( 'divisions_id' => array( 'description' => 'Division code', 'type' => 'varchar', 'length' => '5', 'not null' => TRUE, ), 'div_desc' => array( 'description' => 'Division name', 'type' => 'varchar', 'length' => '35', 'not null' => FALSE, ), ), 'primary key' => array('divisions_id'), );

I created another database table to hold the extra data about divisions.

$schema['hank_divisions_extra_data'] = array( 'description' => 'Extra Drupal data for divisions', 'fields' => array( 'divisions_id' => array( 'description' => 'Division code', 'type' => 'varchar', 'length' => '5', 'not null' => TRUE, ), 'div_url' => array( 'description' => 'Division website', 'type' => 'varchar', 'length' => '128', 'not null' => FALSE, ), 'div_head_id' => array( 'description' => 'Division head', 'type' => 'int', 'not null' => FALSE, ), 'div_office_id' => array( 'description' => 'Division office location', 'type' => 'int', 'not null' => FALSE, ), ), 'primary key' => array('divisions_id'), );

We can include the data from this second table during entity_load() with the attachLoad() method in our controller subclass.

/** * HankDivEntityController extends EntityAPIController. * * Our subclass of EntityAPIController lets us add a few * customizations to create, update, and delete methods. */ class HankDivEntityController extends EntityAPIController { public function attachLoad(&$queried_entities, $revision_id = FALSE) { foreach ($queried_entities as $entity) { $result = db_query('SELECT * FROM {hank_divisions_extra_data} WHERE divisions_id = :divisions_id', array(':divisions_id' => $entity->divisions_id))->fetchAll(); if (!empty($result)) { $data = reset($result); if (!empty($data->div_url)) { $queried_entities[$entity->divisions_id]->div_url = $data->div_url; } if (!empty($data->div_head_id)) { $queried_entities[$entity->divisions_id]->div_head_id = $data->div_head_id; } if (!empty($data->div_office_id)) { $queried_entities[$entity->divisions_id]->div_office_id = $data->div_office_id; } } } return parent::attachLoad($queried_entities, $revision_id); } }

If we added an edit form for this entity, we'd want to override the save() method in our controller to write the extended data back to its table.

Simple db_query() to populate list fields.

So now we have all of these really cool entities and can do lots of stuff with them in Drupal. But sometimes, entity_load() is too slow for our needs, and we just want to get data quickly. As it turns out, db_query() is still the fastest way to get at this data, and we don't really need the entity system at all.

/** * Returns a list of HFC employees for use as field options. */ function staffdir_options() { $options = &drupal_static(__FUNCTION__); if (!isset($options)) { if ($cache = cache_get('staffdir_options')) { $options = $cache->data; } else { $persons = db_query('SELECT hrper_id, first_name, last_name FROM {hank_hrper} ORDER BY last_name, first_name')->fetchAll(); $options = array(); foreach ($persons as $value) { $options[$value->hrper_id] = $value->first_name . ' ' . $value->last_name; } cache_set('staffdir_options', $options, 'cache', time() + 900); } } return $options; }

Note that for added performance, we made this a static variable and cached the results. We're using memcache, so this lookup won't require a database lookup once it's cached.

See also:

Part 2. Providing web services

This section will introduce the Services and Services Views modules, then show how to write custom hook code for better performance.

Services Module

Supports full CRUD functions with authentication. We're only interested in anonymous index and retrieve functionality for this project.

The handbook covers installation pretty well, so I'm not going to cover it here. The quickest way to see what this does for us is to just set up a services endpoint and enable node index and retrieve, then visit

  • http://example.com/path/to/endpoint/node
  • http://example.com/path/to/endpoint/node/1

Services node display

Note: When testing anonymous endpoints from the browser, your session will be logged out. I always work in one browser and test output in another.

See also:

Services Views Module

This is where things really start to get fun! Services View allows you to create a web service resource from a view. After adding the display, be sure to enable it as a resource on the desired endpoint.

Define fields and filters

Here are the fields and filters for my Staff Directory view. Note that in this case, all of the filters are exposed.

Views fields and filters

Set custom labels on fields

If you want to make life easy for yourself, set easy-to-reference field labels on your fields. These will become properties on the retrieved objects in our client. Use underscores, not dashes. We'll see a little later why the dashes used here were a really bad idea.

Services Views field labels

Set custom field identifiers

The identifiers set on exposed filters will become the query arguments for retrieving data. Again, keep these as not ugly things you won't regret having be part of a URL.

Views filter identifiers

Review output

Views fields and filters

See also:

hook_services_resources()

Once again, while all there are tools to make all of this really fun and easy, the overhead of using entities and views can really slow down your site. Let's go back to our divisions table we talked about earlier. Even with our custom fields, it's a pretty simple database query to create a list of these entities.

We start by using hook_services_resources() to define our resource.

/** * Implements hook_services_resources(). */ function hankdata_services_resources() { return array( 'divisions' => array( 'operations' => array( 'index' => array( 'help' => 'Retrieves a list of divisions.', 'callback' => '_hankdata_hankdiv_index', 'access callback' => 'user_access', 'access arguments' => array('access content'), 'access arguments append' => FALSE, ), ), ), ); }

Note that callback must be defined using a private function name starting with an underscore.

Now use db_query() to generate a list of divisions. (You might prefer to use db_select() to keep the code prettier, but db_query() is faster.)

/** * Returns an index of HFC divisions for use as a services endpoint. */ function _hankdata_hankdiv_index() { $query = "SELECT d.divisions_id, d.div_desc, "; $query .= "x.div_url, x.div_head_id, x.div_office_id "; $query .= "FROM {hank_divisions} d LEFT JOIN {hank_divisions_extra_data} x ON d.divisions_id = x.divisions_id "; $query .= "ORDER BY d.divisions_id ASC"; $result = db_query($query)->fetchAll(); return $result; }

TODO: Provide sample output.

See also:

An introduction to Drupal 7 RESTful Services - Alex Rayu at Pingv.

Part 3. Making a web services client

This section will show how easy it can be to retrieve data from another website by taking advantage of the Services module's ability to send serialized PHP arrays and objects. We'll also look at a fake field formatter so your custom data can be themed to look just like regular Drupal fields.

Why not remote entities?

To make a long story short, the biggest problem I kept hitting was that the Drupal 7 Entity system, EntityFieldQuery and the EFQ_Views module would all really prefer a local base table in the database. I started writing custom code to work around that, based on Florian's blog post linked below, but the bottom line was that all I wanted to do was diplay data, so why build remote entities when all I needed was render arrays?

I did, however, use a lot of ideas from Florian's post to write some of my more complicated retrieval functions.

See also:

Remote entities in Drupal 7 - Florian Loretan at Wunderkraut

Serialized arrays and objects FTW!

The Services sends its output in a variety of formats, like XML, JSON, YAML, etc. When talking between Drupal sites, the easiest format is PHP. The data response is simply a serialized object or array of objects.

IMPORTANT UPDATE: It came to my attention after giving this presentation that there are potential vulnerabilities due to the fact that PHP will execute any __wakeup() function, and will execute unserialized_callback_func if encountered when running unserialize() on objects. Use this method only with trusted resources.

UPDATE: Since this presentation, I have switched to JSON, replaced calls to drupal_http_request() with the Guzzle PHP library, and implemented OAuth security. See also, Matt Farina's DrupalCampMI session, Secure Your Site.

/** * Retrieve a URL from the services host. * * @todo Change this to use https://drupal.org/project/http_client */ function hfccwsclient_http_request($request_url) { if (!$api_url = variable_get('hfccwsclient_api_url', NULL)) { drupal_set_message(t('Please configure the web services URL before using this function.'), 'error'); return FALSE; } $response = drupal_http_request($api_url . $request_url); if ($response->code == '200' && !empty($response->data)) { return unserialize($response->data); } else { if (!empty($response->error)) { drupal_set_message(check_plain($response->code . ': ' . $response->error)); } return FALSE; } }

Here is a function that retrieves staff directory entries, with optional arguments:

/** * Retrieve staff directory from the services host. */ function hfccwsclient_get_staffdir($filter_opts = array()) { $args = !empty($filter_opts) ? '?' . implode('&', $filter_opts) : NULL; if ($staffdir = hfccwsclient_http_request('/staffdir.php' . $args)) { $output = array(); foreach ($staffdir as $person) { $output[$person->{'hank-id'}] = (array) $person; } } else { drupal_set_message(t('Could not retrieve staff directory.')); return FALSE; } return $output; }

As you can see, I didn't take my own advise about naming fields coming out of the view, requiring me to escape the hank-id property.

And another that works similarly to retrieve just an options list keyed on Employee ID number:

/** * Retrieve staff directory by HANK IDs as options list. */ function hfccwsclient_get_staffdir_hankid_opts($filter_opts = array()) { $args = !empty($filter_opts) ? '?' . implode('&', $filter_opts) : NULL; if ($staffdir = hfccwsclient_http_request('/staffdir.php' . $args)) { $output = array(); foreach ($staffdir as $person) { $fullname = !empty($person->firstname) ? $person->firstname : ''; $fullname .= !empty($person->lastname) ? ' ' . $person->lastname : ''; $output[$person->{'hank-id'}] = trim($fullname) . ' (' . $person->{'hank-id'} . ')'; } } else { drupal_set_message(t('Could not retrieve staff directory.')); return FALSE; } return $output; }

Both of these can be called with arguments like this:

$args = array("division=NSG", "lastname=smith"); $staffdir = hfccwsclient_get_staffdir($args); dpm($staffdir, 'staffdir data');

Parse retrieved content to Drupal Render Arrays

In some cases, the data on the services host is assembled into render arrays before sending it out via hook_services_resources(), but in most cases, all of the display is handled by the web services client.

Here's a basic example that generates a table of the results:

$staffdir = hfccwsclient_get_staffdir($args); $rows = array(); foreach ($staffdir as $person) { rows[] = array( !empty($person['lastname']) ? $person['lastname'] : '', !empty($person['firstname']) ? $person['firstname'] : '', !empty($person['department']) ? $person['department'] : '', !empty($person['email-address']) ? $person['email-address'] : '', ); } $output = array( '#theme' => 'table', '#header' => array( t('Last Name'), t('First Name'), t('Department'), t('Email Address') ), '#rows' => $rows, '#empty' => t('No results returned'), );

When it comes to displaying individual items with the web services client, it takes a little more work to render them to look like actual Drupal fields.

I'm going to use course listings for this example. Rather than retrieve individual courses from the services host, we pull the entire course detail in one big array and cache it. This does make the initial load a little slow, especially if caches have been cleared on both server and client, but it's super-fast after that, since everything is served out of memcache.

Here's a portion of the code used to build an individual course display.

/** * Build course detail. */ function hfccwsclient_courses_build_detail($id) { $courses = hfccwsclient_get_courses(); if (is_array($courses)) { $course = $courses[$id]; $output = array(); if (!empty($course['division'])) { $output['division'] = array( '#type' => 'item', '#label' => t('Division'), '#label_display' => 'inline', '#theme_wrappers' => array('hfcc_global_pseudo_field'), '#markup' => $course['division'], ); } // // Skip building the other fields for now. // $course_title = $course['subject'] . ' ' . $course['number'] . ' ' . decode_entities($course['title']); return array( 'title' => $course_title, 'body' => $output, ); } else { drupal_set_message(t('An error occurred while retrieving the course listing. Please try again later.'), 'error'); } }

This returns an array with two elements, title and body that can be used in a page callback, or in other ways, like a text filter to display course listings inline with other content.

See also:

hfcc_global_pseudo_field.tpl.php

Drupal doesn't provide a good way to render fake fields using the same HTML that real fields get. you can can force it to use '#theme' => 'field', but that requires setting a bunch of false information.

We've created a template copied from field.tpl.php and template_preprocess_field() and put it in the HFC Global module that we use on all of our sites. The posts below have more detail on that.

See also:

Further reading

Resources

HFC source code on Bitbucket. Fork if you like, but it's all unsupported.

Find me

Edits

  • This post has been updated to reflect the name change from Henry Ford Community College (HFCC) to Henry Ford College (HFC).
Topics: