Alex Moon

There is a kind of exciting movement in Web dev right now. Web devs are talking about "JavaScript Fatigue", "Framework Fatigue", the "Revival of Hypermedia" and "HTML Over The Wire". In a word: we're asking ourselves why we're building HTML in JavaScript.

The figurehead for this movement is undoubtedly HTMX. It shows that much of what we do in JavaScript could instead be done declaratively, with HTML attributes. If browsers adopted these semantics natively, many websites - and even apps - wouldn't need JavaScript at all. I love this idea! Writing HTML first and adding JS on top is the way the Web should work.

At present, we write JavaScript first, and we use it to generate HTML. How did we get it so backwards? I believe the answer is pretty straightforward: SPA frameworks are a joy to use. They impose structure, enforcing conventions, ultimately making it easy to keep concerns separated in one's mind.

My big problem with HTMX, as it stands, is that it lacks that structure. Taking a look at HTMX the first time, my reaction was: "...so, declarative jQuery." I could see, as if before my very eyes, the spaghetti that inevitably grows out of a library like this. HTMX leaves it up to the developer to impose discipline on their code, however they see fit.

So, I decided to accept the challenge. I want to do modular SSR the way HTMX encourages, but I want to do it with something like an SPA framework. I want nestable components, each with their own HTML, CSS, and JS - and back-end code - sitting side by side. I want there to be one, and only one, right way to do something.

The result of this journey is MESH - modular element SSR with hydration. MESH is based on a simple principle: one component = one endpoint. This is a powerful idea - it allows us to write a HTML-first back-end in such a way that it feels like writing an SPA.

This write-up includes a lot of code snippets. I've tried to keep these minimal. If you want to follow along with more context, you can find the whole commit history for MESH on GitHub.

Basic Interactivity

Looking around, it seemed the back-end of choice for HTMX devs is Go with Templ. I've never really had my "Damascus moment" with Go, but this was a good opportunity to get my feet wet. I will say this much: it is a joy to work with something genuinely blazingly fast to build and deploy.

I also wanted to have a go with proper vibe coding - writing code without reading it - with Junie. What fun this was! I can see why people would be tempted to write whole apps this way. I'll only say this much: as someone who's battled addiction in the past, I didn't like what I noticed my brain was doing with it. That's a subject for another blog post another time.

My mission was, in a word, to write something like an opinionated framework or "harness" for HTMX which would give me a standard way to use it with Web Components. Specifically, what I had in mind was a "one component one endpoint" model. HTMX would always swap the entire component, which would then be "hydrated".

It turns out there is a standard way to do server-side rendered custom elements, called Declarative Shadow DOM (DSD). Others have already had some success using HTMX and DSD together. The combo looked promising.

There is one significant limitation, however: HTMX will not cross shadow root boundaries. This is by design, to be clear - this is how we should expect HTMX to behave. No sweat, we can do a simple hack to make it work - and, at the same time, to enforce component-level swaps:

import type {HtmxBeforeSwapDetail} from "./types/htmx";

function enforceComponentSwap(evt: CustomEvent<HtmxBeforeSwapDetail>) {
  const detail = evt.detail;
  let elt = detail.elt;
  let root = elt.getRootNode();
  
  if (root instanceof ShadowRoot) {
    detail.target = root.host as HTMLElement;
    detail.swapOverride = "outerHTML";
  }
}

document.body.addEventListener("htmx:beforeSwap", enforceComponentSwap as EventListener);

With this little helper, I can now start building out a very simple Trello clone to prove the concept. Let's build a little editable card component:

package card

import (
  "mesh/src/services"
  "fmt"
)

type CardProps struct {
  *services.Card
}

templ Card(props CardProps) {
  <mesh-card
    id={ fmt.Sprintf("card-%d", props.Card.ID) }
  >
    <template shadowrootmode="open">
      <base href="/"/>
      <link rel="stylesheet" href="/static/css/components/card.css"/>
      <div data-view class="card">
        <div class="card-header">
          <h3>{ props.Card.Title }</h3>
        </div>
        <div class="card-content">
          { props.Card.Content }
        </div>
        <div class="actions">
          <button type="button" mesh-click="edit">Edit</button>
        </div>
      </div>
      <form data-form class="card hide" hx-patch="/card">
        <input type="hidden" name="cardID" value={ props.Card.ID } />
        <label>
          Title
          <input type="text" name="title" value={ props.Data.Title } />
        </label>
        <label>
          Content
          <textarea name="content">{ props.Data.Content }</textarea>
        </label>
        <div class="actions">
          <button type="button" mesh-click="cancel">Cancel</button>
          <button type="submit">Save</button>
        </div>
      </form>
    </template>
  </mesh-card>
}

Now let's hydrate it. I'm going to start with a simple base element that'll ensure that our shadow root is attached properly and processed by HTMX:

export class MeshElement extends HTMLElement {
  connectedCallback() { 
    if (!this.shadowRoot) {
      // the browser should do this for us - oh well, what can you do?
      const root = this.attachShadow({ mode: 'open' });
      const template = this.querySelector('template[shadowrootmode="open"]');
      if (template) {
        root.appendChild((template as any).content.cloneNode(true));
      }
    }
    if (window.htmx) {
      window.htmx.process(this);
      if (this.shadowRoot) {
        window.htmx.process(this.shadowRoot);
      }
    }
    this.bindListeners();
  }

  protected bindListeners() {
    const supportedEvents = ['click'];

    supportedEvents.forEach(eventName => {
      const attribute = "mesh-" + eventName;
      this.all('[' + attribute + ']', el => {
        const methodName = el.getAttribute(attribute);
        if (!methodName) {
          return;
        }
        const method = (this as any)[methodName];
        if (!method || typeof method !== 'function') {
          console.error(`Method ${methodName} is not a function`);
          return;
        }

        el.addEventListener(eventName, method.bind(this));
      });
    });
  }

  all(selector: string, cb: (el: HTMLElement) => void) {
    return this.shadowRoot!.querySelectorAll(selector).forEach(e => cb(e as HTMLElement));
  }
}

Then our card element is straightforward to implement:

import {MeshElement} from "../base/mesh-element.ts";

export class Card extends MeshElement {
  edit() {
    this.show('[data-form]');
    this.hide('[data-view]');
  }
  
  cancel() {
    this.hide('[data-form]');
    this.show('[data-view]');
  }

  show(selector: string) {
    this.all(selector, el => {
      el.classList.remove('hide');
    });
  }

  hide(selector: string) {
    this.all(selector, el => {
      el.classList.add('hide');
    });
  }
}
window.customElements.define('mesh-card', Card);

This works great! I've enhanced my card component with some basic JS to show that it can be done, and otherwise this is all just plain old HTMX.

The next step is adding functionality to move the cards between columns. This is where we'll run into a common difficulty with HTMX: how to swap out "parent" components given an update on a child component.

Now, HTMX devs have a number of differing opinions on how best to go about this. One common practice is to "expand the target", which means your component needs to be aware of parent components. Another way is to trigger events in the response headers - this is better, in that it moves responsibility for this back to the server. I believe front-end components shouldn't know anything their own placement on the page.

Fortunately, HTMX gives us another way to do this - and it appears to be the emerging "best practice" - with "out of band" (OOB) swaps. If, in our response to a call to the card endpoint, we return any other components that need updating, and simply flag them as OOB, HTMX will handle the swaps for us. This best reflects my own aims for MESH, so let's see how we get along doing it this way.

Let's add "promote" functionality to our card component - this will simply move the card one column to the right:

package card

import (
  "mesh/src/services"
  "fmt"
)

const PutActionPromote = "promote"

type CardProps struct {
  *services.Card
  CanPromote bool
}

templ Card(props CardProps) {
  <mesh-card
    id={ fmt.Sprintf("card-%d", props.Card.ID) }
  >
    <template shadowrootmode="open">
      <base href="/"/>
      <link rel="stylesheet" href="/static/css/components/card.css"/>
      <div data-view class="card">
        <div class="card-header">
          <h3>{ props.Card.Title }</h3>
        </div>
        <div class="card-content">
          { props.Card.Content }
        </div>
        <div class="actions">
          if props.CanPromote {
            <form hx-put="/card">
              <input type="hidden" name="action" value="promote" />
              <input type="hidden" name="cardID" value={props.Card.ID} />
              <button type="submit" aria-label="Move to next column">
                <i data-lucide="arrow-right"></i>
              </button>
            </form>
          }
        </div>
      </div>
    </template>
  </mesh-card>
}

To handle the OOB updates, we'll write a "context-enriched" pub-sub:

type EventContext struct {
	Context        context.Context
	ResponseWriter http.ResponseWriter
}

func (e *EventContext) Write(component templ.Component) {
	err := component.Render(e.Context, e.ResponseWriter)
	if err != nil {
		http.Error(e.ResponseWriter, "Failed to render OOB updates", http.StatusInternalServerError)
	}
}

func (e *EventService) Publish(event Event, w http.ResponseWriter, ctx context.Context) {
  eventContext := EventContext{
    Context:        ctx,
    ResponseWriter: w,
  }
  for _, subscriber := range e.subscribers[event.Key()] {
    subscriber(event, eventContext)
  }
}

func (e *EventService) Subscribe(key string, subscriber func(event Event, context EventContext)) {
  e.subscribers[key] = append(e.subscribers[key], subscriber)
}

Then we can publish in the card handler:

func (h *Handler) Put(w http.ResponseWriter, r *http.Request) {
  card, err := h.getCardFromRequest(r)
  if err != nil {
    http.Error(w, err.Error(), http.StatusNotFound)
    return
  }
  action := r.FormValue("action")
  switch action {
  case PutActionPromote:
    fromColumn, toColumn, err := h.CardService.Promote(card.ID)
    if err != nil {
      http.Error(w, err.Error(), http.StatusBadRequest)
    } else {
      h.EventService.PublishCardMoved(card.ID, fromColumn.ID, toColumn.ID, w, r.Context())
    }
    break
  }
}

And we can subscribe in the column handler:

func (h *Handler) OnCardMoved(event *services.CardMovedEvent, context services.EventContext) {
  column, err := h.CardService.GetColumn(event.ToColumnID)
  if err == nil {
    context.Write(h.RenderComponent(column, true))
  } else {
    http.Error(context.ResponseWriter, err.Error(), http.StatusInternalServerError)
  }
  column, err = h.CardService.GetColumn(event.FromColumnID)
  if err == nil {
    context.Write(h.RenderComponent(column, true))
  } else {
    http.Error(context.ResponseWriter, err.Error(), http.StatusInternalServerError)
  }
}

This way, our components can communicate with each other without needing to know about each other. The subscriber takes the request context from the publisher and simply writes to the response. The result is a response from the back-end with

1. the component-specific update, followed by
2. any other OOB updates simply appended to the response.

This works surprisingly well.

Unfortunately, we once again run into the same limitation as before: HTMX will not cross shadow root boundaries. At this point, it seems clear that, if we want to use HTMX as intended, we are going to have to give up on shadow DOM entirely. Again, this is intended behaviour. JavaScript should not cross shadow root boundaries by default. What this means is that, if we're wedded to shadow DOM, we are going to have to fight HTMX all the way.

I'm undeterred, of course - all we need is another little hack:

function findInShadow(root: any, id: string): any {
  const element = root.getElementById?.(id);
  if (element) {
    return element;
  }
  const allElements = root.querySelectorAll('*');
  for (let element of allElements) {
    if (element.shadowRoot) {
      const found = findInShadow(element.shadowRoot, id);
      if (found) {
        return found;
      }
    }
  }
  return null;
}

function enableOobSwap(evt: CustomEvent<any>) {
  const id = evt.detail.content.id;
  const found = findInShadow(document, id);
  if (found) {
    found.outerHTML = evt.detail.content.outerHTML;
    evt.preventDefault();
  }
}

document.body.addEventListener("htmx:oobErrorNoTarget", enableOobSwap as EventListener);

You'll notice we've done the outerHTML swap ourselves here, overriding HTMX entirely. I'm not a fan of this. I tried as many approaches as I could think of to get HTMX to do the swap - and, thus, leverage existing functionality HTMX provides for these (maintaining scroll position and focus and so on) - but did not succeed.

Nevertheless, this works for my purposes. I'm happy to leave the problem alone for now. Let's add some drag-and-drop functionality to our cards. In our card component:

import {MeshElement} from "../base/mesh-element.ts";

export class Card extends MeshElement {
  setupDragAndDrop() {
    this.one('.grip', grip => {
      grip.draggable = true;
      this.addEventListener('dragstart', this.handleDragStart.bind(this));
      this.addEventListener('dragend', this.handleDragEnd.bind(this));
    });
  }

  handleDragStart(e: any) {
    e.dataTransfer.setData('text/plain', this.dataset.id);
    this.classList.add('dragging');
    e.dataTransfer.effectAllowed = 'move';
  }

  handleDragEnd() {
    this.classList.remove('dragging');
  }
}

And in our column component:

import {MeshElement} from "../base/mesh-element.ts";

export class Column extends MeshElement {
  setupDropTarget() {
    this.addEventListener('dragover', this.handleDragOver.bind(this));
    this.addEventListener('drop', this.handleDrop.bind(this));
  }

  handleDragOver(e: any) {
    e.preventDefault();
    e.dataTransfer.dropEffect = 'move';
  }

  handleDrop(e: any) {
    e.preventDefault();
    this.classList.remove('drag-over');

    const cardId = e.dataTransfer.getData('text/plain');
    const columnId = this.dataset.id;
    if (!cardId || !columnId) {
      throw new Error('Missing card or column ID');
    }

    const position = this.calculateDropPosition(e);
    this.moveCard(cardId, +columnId, position);
  }

  async moveCard(cardId: number, columnId: number, position: number) {
    window.htmx.ajax('put', '/card', {
      swap: 'none',
      values: {
        action: 'move',
        cardID: cardId,
        columnID: columnId,
        position: position,
      }
    } as any);
  }
}

This is great! This use case is precisely why HTMX provides the ajax JS API. With a bit of hacking, we've demonstrated that it's possible to use HTMX to handle modular SSR based on the premise of "component = endpoint". I'm pretty happy with how this has turned out.

Realtime Collaboration

From the moment I conceived of this project, one of the things I wanted to do was to support realtime collaboration with server-sent events (SSE). HTMX supports SSE with a standard plugin which is easy enough to set up:

<html lang="en" hx-ext="sse">
<body hx-ext="sse" sse-connect="/sse" sse-swap="oob-update">

In theory this should just work once I've written my SSE back-end. I ended up using r3labs/sse for this, which I found very easy to use. We wrap this in a service and provide a "broadcast" method that sends OOB updates to all subscribed clients immediately:

func (s *SSEService) BroadcastOOBUpdate(component templ.Component) {
   var buf strings.Builder
   err := component.Render(context.Background(), &buf)
   if err != nil {
       s.log.Error("Failed to render component for SSE broadcast", "error", err)
       return
   }

   html := buf.String()
   s.server.Publish("oob-updates", &sse.Event{
       Event: []byte("oob-update"),
       Data:  html,
   })
}

Then we call it in our handler:

func (h *Handler) OnCardMoved(event *services.CardMovedEvent) {
  column, err := h.CardService.GetColumn(event.ToColumnID)
  if err == nil {
    component := h.RenderComponent(column, true)
    h.SSEService.BroadcastOOBUpdate(component)
  } else {
    h.Log.Error("Failed to get to-column for SSE broadcast", "columnID", event.ToColumnID, "error", err)
  }

  column, err = h.CardService.GetColumn(event.FromColumnID)
  if err == nil {
    h.SSEService.BroadcastOOBUpdate(h.RenderComponent(column, true))
  } else {
    h.Log.Error("Failed to get from-column for SSE broadcast", "columnID", event.FromColumnID, "error", err)
  }
}

This makes our back-end code a lot cleaner! We no longer need to pass the request context around with our event, and we no longer need to append a bunch of OOB updates to the response. Having done this both ways, I have come to believe that SSE is the most natural way to do these kind of asynchronous cross-context modular updates, even with only a single user.

I was hoping this would also allow me to get rid of my outerHTML hack, but alas it was not to be. The longer I worked on this project, the more it became clear to me that I'm not really using HTMX the way it's intended to be used. More importantly, there is a lot of other HTMX functionality that I'm not using at all.

Naturally, I was intrigued to see if I could just get rid of HTMX entirely. So I did, and the result is a lot cleaner and easier to reason about. We are left with two JS modules - one for the custom elements:

export class MeshElement extends HTMLElement {
  connectedCallback() {
    if (!this.shadowRoot) {
      const root = this.attachShadow({ mode: 'open' });
      const template = this.querySelector('template[shadowrootmode="open"]');
      if (template) {
        root.appendChild((template as any).content.cloneNode(true));
      }
    }
    this.bindFormHandlers();
  }

  protected bindFormHandlers() {
    const supported = [
      'get', 'post', 'put', 'patch', 'delete',
    ];

    supported.forEach(verb => {
      const attribute = "mesh-" + verb;
      this.all('[' + attribute + ']', el => {
        const form = el as HTMLFormElement;
        form.addEventListener('submit', (event: Event) => {
          event.preventDefault();
          const method = verb.toUpperCase();
          const url = form.getAttribute(attribute);

          if (!url) {
            console.error('No URL specified for form submission');
            return;
          }

          const formData = new FormData(form);
          this.makeRequest(method, url, formData)
            .then(response => {
              if (response.ok) {
                return response.text();
              } else {
                throw new Error('Form submission failed: ' + response.statusText);
              }
            })
            .then(html => this.outerHTML = html)
            .catch(error => console.error('Form submission failed:', error));
        });
      });
    });
  }

  protected async makeRequest(method: string, url: string, formData: FormData): Promise<Response> {
    const options: RequestInit = {
      method,
      headers: {
        'X-Requested-With': 'XMLHttpRequest',
      },
    };

    if (method === 'GET') {
      const params = new URLSearchParams(formData as any);
      url += (url.includes('?') ? '&' : '?') + params.toString();
    } else {
      options.body = formData;
    }

    return fetch(url, options);
  }
}

and one for SSE:

export class SSEManager {
  private eventSource: EventSource | null = null;

  constructor(private url: string = '/sse?stream=oob-updates') {
    this.connect();
  }

  private connect() {
    if (this.eventSource) {
      this.eventSource.close();
    }

    this.eventSource = new EventSource(this.url);

    this.eventSource.addEventListener('oob-update', (event) => {
      this.processOOBUpdate(event as MessageEvent);
    });

    this.eventSource.onerror = (error) => {
      console.error('SSE connection error:', error);
      setTimeout(() => this.connect(), 5000);
    };
  }

  private processOOBUpdate(html: string) {
    const template = document.createElement('template');
    template.innerHTML = html.trim();

    for (const content of template.content.querySelectorAll('[mesh-swap-oob]')) {
      const id = content.id;
      const target = this.findInShadow(document, id);

      if (target) {
        target.outerHTML = content.outerHTML;
      } else {
        console.warn('OOB target not found:', id);
      }
    }
  }

  private findInShadow(root: Document | ShadowRoot | Element, id: string): Element | null {
    let element = root.querySelector(`#${id}`);
    if (element) {
      return element;
    }

    const allElements = root.querySelectorAll('*');
    for (const el of allElements) {
      if (el.shadowRoot) {
        element = this.findInShadow(el.shadowRoot, id);
        if (element) {
          return element;
        }
      }
    }

    return null;
  }
}

new SSEManager();

And that's it! That's all the JS it takes to replace all of HTMX that I'm using for this project.

Takeaways

This was a fun project. First, let me say, if you're writing apps with jQuery, please check out HTMX! It's very dev-friendly and a proper 2020s way of doing that kind of dev. Personally, however, I am happy to have convinced myself it's not for me.

I, for one, don't believe the HTMX spec, or something like it, will be merged back into HTML, at least until it can answer one fundamental question: what is the default swap behaviour? When I declare a form with a method, I understand how that form will behave: it will reload the entire page. What happens when I declare a form with hx-post or equivalent? The default behaviour in HTMX is that the innerHTML of the form itself becomes the swap target. This does not seem like a sane default to me.

So what's the answer? Well, as anyone who's aware of the state of the art on SSR will have noticed, all I've actually done with MESH is reinvent HotWire, LiveWire, LiveView and friends. Personally, I find this encouraging! It is clear to me that there is a kind of best practice to be found here.

I believe the default swap behaviour should be: always swap the whole component. One component, one endpoint. This is how these frameworks do it. My problem with them is they lock you into a specific back-end. I believe the principle is generalisable, that there is a way to do this kind of modular SSR in a back-end-agnostic way, like HTMX does. MESH is my attempt to show what that would look like.

I will certainly keep using MESH for my projects, fleshing it out as I go. The Trello clone will always be online for anyone to play with.