Decoupled Drupal with Silex

Presented by Larry Garfield (@Crell) / Hagen Last

@Crell

Palantir.net Pays me, too!
  • Senior Architect, Palantir.net
  • Drupal 8 Web Services Lead
  • Drupal Representative, PHP-FIG
  • Advisor, Drupal Association
  • Loveable pedant

Hagen Last

  • Sr. Solutions Architect, Ooyala Pays me, too!

About Ooyala Pays me, too!

  • Leading OVP provider
  • Founded in 2007
  • 300+ employees worldwide
  • Global footprint of 200M unique users in 130 countries
  • Ooyala works with the most successful broadcast and media companies in the world

Ooyala Customers

Major Media Companies

CMS for OTT Solution

This is what we need to build
  • Support for structured metadata
  • UI to manage metadata
  • Publishing workflow
  • Curate content
    • Lists, Collections
    • Banners (Hero)
    • Homepage and lists of lists

...and

  • Data sync with ooyala backlot
  • APIs that work at scale
    • Thousands of movies
    • Tens of Thousands of users
    • Support website and applications

Team Palantir

You know what would be an awesome
platform for this project?

Drupal 8 would be perfect!

Oh, right, 2013...

So now what?

  1. Drupal 7 with Services
  2. Drupal 7 with RestWs
  3. Symfony
  4. Silex

Why choose?

Drupal 7

Drupal 7

Pro: CMS and page display

Con: Mediocre at REST

Silex

Silex

Pro: HTTP handling

Con: Hand-rolled artisinal UI

Drupal 7 Silex

Communication?

  • Drupal data is Drupal's
  • Editorial structure != API structure

Answer

Elasticsearch
XML file
=>
Drupal 7
=>
Elasticsearch
=>
Silex
=>
Client applications

The data model

(aka The Nodes)

Major content

  • Program
  • Asset
  • Offer
  • Collections
  • Various app-specific content

Where does the data
come from?



  
The Dark Night When Batman, Gordon and Harvey Dent... Movie:Action/Adventure Christian Bale

Importing

  • Migrate
  • Feeds
  • Services
  • Custom

Importing

QueryPath, an XML parsing library
+

Closure-based mapping objects

=

Potentially multiple nodes

HTTP

  • Cheat: POST-only, no REST
  • 3 page callbacks, common code
  • Vanilla Drupal
  • Http Basic (over SSL) (simplehttpauth)

One more thing…

Rotten Tomatoes
  • Guzzle
  • Guzzle lib for Rotten Tomatoes
  • Rotten Tomatoes module
  • Map nodes by IMDB ID
  • Refetch periodically

Editing

Drupal's got this, right?

...Almost

Content index

The content index page

Editorial content

Program editing

Program editing

Program editing

Program editing

Program editing

Program editing

Where does the data go?

Export to ElasticSearch

  • Lots of preprocessing needed
  • Existing contribs weak (2013)
  • Rules has too many moving parts
  • Exporting some nodes requires others

Another mini-custom OOP system!

When?

  • Published == Public
  • Public == In Elasticsearch
  • Published == In Elasticsearch

So when should it be public?

Publication rules

A Program is publishable when:

  • It is curatorially approved

Actually it's a bit more complex than that

Publication rules

An Offer is publishable when:

  • It is curatorially approved
  • … and the Offer is within its Publication window
  • … or it's about to be

Actually it's a bit more complex than that

Publication rules

An Asset is publishable when:

  • It is curatorially approved
  • … and it has an Offer that is approved
  • or the Offer is within its Publication window
  • … or it's about to be

Actually it's a bit more complex than that

Publication rules

It's complicated…

…And potentially expensive

Cron & Queues

The somewhat involved pipeline for indexing content
Cron->Publish queue: Ready nodes Cron->Publish queue: Related nodes Publish queue->Node save: Publish Cron->Publish queue: Expiring nodes Cron->Publish queue: Related nodes Publish queue->Node save: Unpublish Node save->Index queue: Always opt node published Index queue->Index queue: Prepare data Index queue->Elasticsearch: PUT end opt node unpublished Index queue->Elasticsearch: DELETE end

Serving the API

Silex
  • Lightweight microframework
  • Same Kernel/Routing as Symfony (and Drupal 8)
  • Bare-bones, add what you need
    • Elastica (Elasticsearch)
    • Guzzle (talk back to Ooyala)
    • Nocarrier/Hal (the API format)
What are you doing, Drupal?
  • Hypertext Application Language
  • IETF Draft
  • JSON and XML (but who uses the XML version?)

Silex overview

The generic path of data through the system.
Client->+Kernel: Request Kernel->+Controller: Routing Controller->+ESRepository: Load ESRepository->-Controller: Controller->+Formatter: toHal() Formatter->-Controller: Controller->-Kernel: Return Kernel->+View Listener: View Listener->View Listener: Render to Response (JSON or XML) View Listener->View Listener: HTTP caching View Listener->-Kernel: Kernel->-Client:

The HAL Browser

The Hal Browser index page

Program resource

The Hal Browser, program

Person resource

The Hal Browser, person

High availability?

  • Silex is stateless: Spin up several, load balance
  • Elasticsearch clusters easily (admin story)
  • Varnish caching = FAST!

Fast forward a year...

Works on PC/Mac Works on Android/iOS Works on Chromecast

One API, Palantir uninvolved

And growing

  • Highly reliable
  • Library of thousands of assets
  • Thousands of subscribers
  • New features added regularly; architecture held up

Larry Garfield

Palantir.net

Let's make something good together

Keep tabs on our work at @Palantir

Want to hear about what we're doing?

Hagen Last

Ooyala

The World is Watching