Programming

Improving ‘time to first load’ and generating critical CSS for Drupal and WordPress sites

This document is a work in progress….

One thing that we want to fix is ‘time to first load’. As you probably know, people are recommending to put as little in the way of initial loading as possible, and zero external requests. This means no blocking JS in head, and as little CSS as possible, placed inline instead of in an external file – the so-called ‘critical.css’

Our situation was very much the opposite. Most of our sites are on Drupal or WordPress, with quite a few plugins, which generate a lot of CSS and JS. For wordpress, we often use a theme called Divi, which is quite powerful, but it generates a LOT of css and js, and a lot of it we don’t need.

We started by working on CSS, and then moving onto JS.

CSS

  • Using a web inspector, like Firebug or Safari’s developer tools, take a look at your <head> element. Copy all of the internal CSS files and inline CSS you see there into a single stylesheet (External css, like webfont importing, leave for the moment). If a css file has an @import statement from another file, you have to replace the @import statement with the contents of that file. For example – our WordPress main theme style.css has an import statement from Divi style.css. You will now have a VERY big single css file – lets call it style1.css
  • There are some tools you can use to make the CSS file a bit more manageable. I would recommend first to use a tool like CSS Compressor with Moderate settings so that the file is still readable.
    Then, as much as you can, you can set about removing unused css and combining ‘duplicate’ and overriding css statements that were formerly in different files. You dont need to aim for absolute precision here; if you are not sure about some css, you can just leave for later. This applies especially if you are not familiar with the theme. There are some command line tools and paid online tools that can help you with removing unused and duplicate CSS.
  • Now that you have a single file, you can use a free critical css generator, which will generate some critical css. You should save this into a seperate file called critical.css
  • Then you remove all of the css out of style1.css that is already inside critical.css. Diffmerge for Mac helped with this: you can export a ‘diff’ which unfortunately contains some extra characters but you can remove with search/replace. This file is now your new style1.css.

Now we want to put the contents of critical.css inline in the header, and place style1.css in the footer just before all the footer js, and remove all of the other CSS.

  • In WordPress, you can disable all theme CSS in your function.php file: See this solution on stack overflow (Note: we may need to modify this for logged-in users). You can then override the header.php file in your theme, and add a php include statement for the critical.css file just above wp_head() to load the contents directly
    <style type="text/css"><?php include get_stylesheet_directory() . '/.critical.css'; ?></style>
    Note that if you have an external CSS like webfont, for now it is good to just place there manually.
    Then you should place the style css file manually in your footer.php file just above wp_footer. It is probably best to rename this to style.css and make it your theme style file, complete with theme info on top.
    <link rel="stylesheet" href="<?php include get_stylesheet_directory() ?>./style.css" type="text/css" media="all">
  • In Drupal 7 you can also disable css in your theme template.php file, and load certain css for admins

/* Unset all the base drupal stylesheets */
function themename_css_alter(&$css) {
global $user;
$admincss = array(
drupal_get_path('module', 'user') . '/user.css',
drupal_get_path('module', 'contextual') . '/contextual.css',
drupal_get_path('module', 'ckeditor') . '/css/ckeditor.css',
drupal_get_path('module', 'admin_menu') . '/admin_menu.css',
drupal_get_path('module', 'admin_menu') . '/admin_menu.uid1.css',
drupal_get_path('module', 'domain_admin') . '/domain_admin.css'
);
$temp = $css;
foreach($temp as $key => $data) {
if(!$user->uid || !in_array($key, $admincss)) {
unset($css[$key]);
}
}
Then you can override html.tpl.php and paste the critical css file above the $styles variable.

<style><?php include drupal_get_path('theme', 'yourtheme') . '/critical.css'; ?></style>

and the style css file at the bottom above the $page_bottom variable

<link rel="stylesheet" href="<?php include drupal_get_path('theme', 'yourtheme') ./style.css" type="text/css" media="all">

Cleaning up icon fonts

Some themes (eg wordpress Divi) has its own icon font or import a common font, which is pretty big! We should use fontello to create our own much smaller font based on the icons you actually use and import this one instead.

Javascript

more to come…

Searching for large files on your server

Here’s a command that will check the root / folder recursively for files greater than or equal to 1GB and sort the list.

cd / ; du -h | grep '^[0-9]\+\.\?[0-9]\?G' 2>/dev/null | sort -nrk 1

Thanks to Phil from WiredTree for sharing that info, I’m just putting it down here so I can remember it in future 🙂

Redirect rules to change underscore to hyphen

Unfortunately Apache redirect only allows for 9 placeholders, so that’s the maximum amount of undercore changes that this code allows. We are using this mainly within WordPress sites, so here is the entire wp htaccess file, so you know where to position the code.

# BEGIN WordPress

RewriteEngine On
RewriteBase /

# BEGIN underscore to hyphen redirect code
RewriteRule ^([^_]*)_([^_]*)_([^_]*)_([^_]*)_([^_]*)_([^_]*)_([^_]*)_([^_]*)_([^_]*)$ /$1-$2-$3-$4-$5-$6-$7-$8-$9 [R=301,L]
RewriteRule ^([^_]*)_([^_]*)_([^_]*)_([^_]*)_([^_]*)_([^_]*)_([^_]*)_([^_]*)$ /$1-$2-$3-$4-$5-$6-$7-$8 [R=301,L]
RewriteRule ^([^_]*)_([^_]*)_([^_]*)_([^_]*)_([^_]*)_([^_]*)_([^_]*)$ /$1-$2-$3-$4-$5-$6-$7 [R=301,L]
RewriteRule ^([^_]*)_([^_]*)_([^_]*)_([^_]*)_([^_]*)_([^_]*)$ /$1-$2-$3-$4-$5-$6 [R=301,L]
RewriteRule ^([^_]*)_([^_]*)_([^_]*)_([^_]*)_([^_]*)$ /$1-$2-$3-$4-$5 [R=301,L]
RewriteRule ^([^_]*)_([^_]*)_([^_]*)_([^_]*)$ /$1-$2-$3-$4 [R=301,L]
RewriteRule ^([^_]*)_([^_]*)_([^_]*)$ /$1-$2-$3 [R=301,L]
RewriteRule ^([^_]*)_([^_]*)$ /$1-$2 [L,R=301]
# END underscore to hyphen redirect code

RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]


# END WordPress

Show all Draft Posts Plugin for WordPress MU

This plugin will show the draft posts created in all your WPMU blogs. To activate, just download the PHP file and put it in your wp-content/mu-plugins folder. The file can be downloaded here….

To view all posts, click on ‘Site Admin’ and then on the ‘Posts for moderation’ submenu.

To do: make ‘view’ and ‘edit’ links, a bit like the WordPress ‘manage’ interface actually. In any case, it’s probably best if each theme you supply has an ‘Edit this post’ link, so the moderator can first view the post and then edit it if necessary without going back to admin screen.

Archetypes and ATContentTypes, part 2: Templates and customization slots

We will now look at how these schemas are called by the edit templates, and point out slots and hooks which Archetypes have left for customization.

Let us return to our Image type: selecting this type in portal_types and looking at properties will show that atct_edit is the default editing form, as it is for all ATContent Types. There is only one line in atct_edit, calling the master macro in the base_edit template. The main function of this macro is to call other macros for different parts of the page: most of these macros are located in the edit_macros file"

  • the top slot gets filled by edit_macros/topslot
  • javascript is looked after by archetypes_custom_js/macros/javascript_head
  • css: edit_macros/css
  • header,description, body,footer all in edit_macros.

So, let’s take a closer look at some of the macros in edit_macros:

Macros in edit_macros

  • header generates document actions like print, rss ect via the document actions macro and generates a title for the edit form with the name ‘Edit (Content Type)’
  • byline generates the byline, via the document_byline macro
  • archetypes_schemata_links: not sure what this bit does yet 🙂
  • typedescription returns the description of the content type, which will appear at the top of the page under the title. It can be edited in portal_types without customizing, but the resulting edit will only appear in the ‘add item’ listng and not the edit form where you want it.
  • body: This is where the main action happens: for us, the interesting part of this macro is:
<metal:block define-slot="extra_top" />

        <metal:block define-slot="widgets">
          <tal:fields repeat="field fields">
            <metal:fieldMacro use-macro="python:here.widget(field.getName(), mode='edit')" />
          </tal:fields>
        </metal:block>

        <metal:block define-slot="extra_bottom" />

If we look at the above piece of code, it is essentially asking us to go through each field of the image schema and get its widget (For more on schemas and widgets, see part 1). Note the two slots ‘extra-top’ and ‘extra-bottom’ if you want to put something above or below the image widgets without changing the template.

The body macro also contains a good bit to do with sending the form, and the buttons. It contains a block defining Previous, Next (these automatically appear if there is more than one page in the edit form) Save and Cancel buttons. There is also a slot called ‘extra-buttons’ if you need to make any more buttons for your form, so you can make them in another template.

Whilst Archetypes does have a range of customization slots, one cannot make any easily reversible customizations to the schema as one might with a page template, as the files containing the schema are not stored on the zodb.

One is then left with the prospect of defining a new Archetype derived from your old one, either in a new product or as part of an existing one. Then in Extension/Install.py the portal_types tool can be manipulated to use the new type, or alternatively you can set that the old type can be just not globally allowed while the new type would be. Before you run into that course of action, there might be extra hooks for customization I havent found: for example, Villiam pointed out a post_validate hook for validators.

I am very grateful to Villiam Segeda, our resident Archetypes expert, for teaching me what little I know.

Archetypes and ATContentTypes, part 1: The Archetype Schema

All the default types in Plone – file, image, folder – are part of the ATContentTypes product, and are examples of archetypes. Archetypes were integrated into Plone 2.1, and allow you to create content types easily by providing a range of readymade tools so you can generate edit forms without using any HTML.

An example will perhaps best explain how Archetypes works – lets look at the familiar ‘image’ type. The edit form of an Archetype is generated using a schema, and the schema for image can be found in Products/ATContentTypes/content/image.py: here we’ve taken out some of the code to focus on the essentials:

ATImageSchema = ATContentTypeSchema.copy() + Schema((
  ImageField('image',
        required=True,
                ....(lots of other stuff here, to do with image sizing and such).....
        validators = (('isNonEmptyFile', V_REQUIRED),
                      ('checkImageMaxSize', V_REQUIRED)),
        widget = ImageWidget(
                   #description = "Select the image to be added by
                                   clicking the 'Browse' button.",
                   #description_msgid = "help_image",
                   description = "",
                   label= "Image",
                   label_msgid = "label_image",
                   i18n_domain = "plone",
                   show_content_type = False,)),

  ), marshall=PrimaryFieldMarshaller()
  )

That’s all a bit of a mouthful, so we’ll break it up and isolate the essential parts:

ATImageSchema = ATContentTypeSchema.copy() + Schema((
     ImageField(

Every schema contains a collection of fields; in the edit form the short name is one field , the Title is another and so on… The reason we don’t see the title, id, or description in the above schema is they are already defined in ATContentSchema (they are common to all types); the first line of the code joins ATContentTypeSchema to the schema we are creating above.

For an image, there is only one extra field, the ImageField. Lets say we want another field to specify the camera the image was taken with; our schema will then look like:

ATImageSchema = ATContentTypeSchema.copy() + Schema((
      ImageField(   # all the stuff above
           ),

       StringField('camera',
                  required=1,
                  widget=TextAreaWidget(),),

   ), marshall=PrimaryFieldMarshaller()
       )

Archetypes provides many different kinds of field you can include – TextField, BooleanField (for yes/no values) ect. In plone.org there is a list of available Archetypes fields….

OK, lets look at the arguments of ImageField:

ImageField('image',
       required=True,
               ....(lots of other stuff here).....
       validators = (('isNonEmptyFile', V_REQUIRED),
                     ('checkImageMaxSize', V_REQUIRED)),
       widget = ImageWidget(
                                # whatever goes here we'll explain
                                # when we talk about widgets
        )),

 )

The first argument ‘image’ is the title of the field, the required=True puts the little red ‘required’ dot beside the title. Now, the next two are important:

  • validators: these are classes which check that whatever we’ve entered into the field satisfies a set of conditions, and it won’t accept it if it doesn’t. For example, ImageField calls instances of two validator classes: one of them makes sure the uploaded file isn’t empty, and the other rejects the uploaded file if it is too large. There is some readymade validators at Products/validation/validators, although quite often people have the need to add extra ones of their own.
  • widget: the widget is the graphical layout that the field will take; it defines how to render the field into HTML/XML. For example a StringWidget will set out a one-line box; a TextAreaWidget will set out a bigger box, like what is used for the Description. On plone.org you can see pictures of all the available Archetypes widgets ; it is very unlikely you will need to make your own. In many cases, the field type suggests what widget type to use: an ImageField will use an ImageWidget, for example.

The arguments of ImageWidget are quite straightforward:

widget = ImageWidget(
       #description = "Select the image to be added by clicking the 'Browse' button.",
       #description_msgid = "help_image",
       description = "",
       label= "Image",
       label_msgid = "label_image",
       i18n_domain = "plone",
       show_content_type = False,)),

The _msgid label is used by the internationalization (i18n) mechanism to generate translations in other languages.

Ok, one final thing to deal with, the marshaller, this makes Archetype fields understandable to clients such as WebDAV. As you can see above, the schema took two arguments, the field and the marshaller.

Schema (    (ImageField( all the stuff we discussed ),),
                                    marshall=PrimaryFieldMarshaller()
     )

PrimaryFieldMarshaller is the most common marshaller used, apparently.

Ok, so that’s the first part of my ATContentTypes exploration; the second part will deal with the question I first asked when I saw a type schema: how does all this called by a template and end up on our screens?

More unearthed tips….

Again from my email clear out, I have found a few more tips, this time suggested by Priyadarshan…..

Drag and drop item reordering in Plone 2.5

You can just reorder the order items have in a folder just by dragging and dropping them around. This is of course of great usefulness, since the order in a folder dictates the order in the navigation.

Making a static copy of a Plone Site

This is an excellent recipe to make a static copy of a Plone site:
http://www.zopelabs.com/cookbook/1103609775

Searching for Code in Google

http://www.google.com/codesearch allows searches for source code
on the web. Allows filtering for programming language, license etc.

Lisp and dinosaurs

I really like Philip Greenspun’s contributions to the developer’s community and Internet world at large. His posts are usually witty and have about them a sharp vision that is quite rare to find.

Here is a post I liked about LISP. It is not really concerning Plone, but I think it concerns us as seekers of beauty and simplicity in our project.

From "Lisp diehards = Holocaust deniers"

Hmm… it seems that the “Java = SUV of programming languages” posting has stirred up a bit of controversy over at Slashdot and right here on this server. Some people read it as a personal endorsement of PHP, VB, and other semi-baked programming languages. Actually my personal preference is a much darker, uglier, and more shameful secret: Common Lisp, CLOS, plus an ML-like type inferencing compiler/error checker (with some things done in a sublanguage with Haskell semantics and Lisp syntax). Common Lisp dates from around 1982 and ML from 1984.

I try to keep this preference concealed from young people who’ve been raised on a diet of C, Java, C#, Perl, etc. They just wouldn’t find it credible that 20-year-old systems and ideas are actually better than the latest and greatest from Microsoft and Sun.

Imagine my delight in running into a friend yesterday. She’s a 23-year-old graduate student in computer science at Harvard. Conversation rolled around to programming tools. Unprompted she said “What I think would be best is Common Lisp Object System with a modern type system”. I was stunned. I thought it was only dinosaurs like me that clung to Lisp.

I had a second ephiphany for the week… Believing that Lisp circa 1982 plus some mid-1980s ML tricks thrown in is better than all of the new programming tools (C#, Java) that have been built since then is sort of like being a Holocaust denier.

It is also interesting to read the numerous comments to that post:

http://blogs.law.harvard.edu/philg/2003/09/22/lisp-diehards-holocaust-deniers/