Tag Controller

Todones

  • peekMemResponse(): when grab the tag response from tagCache: lookupRsp.first: need to review it to match the CheriTagResponse returned in MultiLevelTagLookup.bsv

Q&A

Who calls this and what is the input?

How does it get feedback from memory?

Where does this being connected?

Connect to l2 cache; and connect to memory to provide proxied memory interface.

The Connection:

L2Cache — tagController — main memory

  • in Memory.bsv: mkMIPSMemory(…):

    • mkConnection(l2CacheMemory, tagController.cache); // tagController cache is connected with l2 cache’s memory interface. They share the same memory interface. SO there WILL be conflicts and thus performance overhead.
    • memory = topMemIfc = tagControllerMemory; // Master/client interface to main memory module (not included in .bsv)
    • see Memory.bsv

  • in MIPSTop.bsv

    • exports a generic memory client interface: memory = theMem = mkMIPSMemory, see MIPSTop.bsv

What is multi-flit transactions? 4-flit transaction?

  • in MemTypes.bsv: CapsPerFlit is used to count how many tagged bits in one flit. One flit is one data in CheriMemoryResponse/Request. see MemTypes.bsv
  • But still, what is a flit here? how it is related with the capability, data cache line, and tag cache line?
    • One flit in this project means one chunk of bus-width data got transmitted via the memory-cpu bus. see MemTypes

How this is connected with data/instruction caches?

  • It does not connect with d/iCache, only connects to the master interface of L2Cache

How does it handle tags of every memory addr?

  • A submodule called MultiLevelTagLookup, see MultiLevelTagLoopup, or TagLookup
  • One memory access request from L2Cache will generate 2 sub-requests: data request and tag request
  • Data request goes to traditional memory interface
  • tag request goes to another layer of cache interface: tagLookup.cache
  • If tag cache hit, the tag will be parsed and being written into the memory request response struct and sent back to l2cache; in this case, one memory request from l2cache, will only generate one request in main memory.
  • If tag cache miss, then, another memory request will be put ot the traditional memory interface. In this case, one memory request from l2cache, will generate two memory request to the main memory.

How does it distinguish tag memory response vs regular memory response ?

CheriMemRequest/Response contains a masterID, the value is used to distinguish which level of cache the request is issued and which level of cache the response is for. In addition, at tagController, it is used to distinguish a regular memory response or a tag response.

  • CheriMemRespone is distinguished at Master response interface, and dispatched either back to L2Cache or back to tagLookup.

  • masterID for tag request/response is first initialized when a tag cache miss happens and a new CheriMemRequest is crafted. see function CheriMemRquest craftTagReadReq/craftTagWriteReq in cherilibs/trunk/MultiLevelTagLookup.bsv;

  • Every CacheCore derived module has the cacheID as the masterID.

  • tagController has the same masterID = 12 as its submodule tagLookup; a request contains this masterID means a tag request/response;

TagController provides a proxy for memory accesses which adds support for tagged memory.

  • Tag values are stored in memory (currently at the top of DRAM)

  • There is a cache of 32ki tags (representing 1MB memory) stored in BRAM (SRAM in FPGA?).

  • Read responses are amended with the correct tag value and write requests update the value in the tag cache (which is later written back to memory)

    // cherilibs/trunk/TagController.bsv
/******************************************************************************
* mkTagController
*
* This module provides a proxy for memory accesses which adds support for
* tagged memory. It connects to memory on one side and the processor/L2 cache
* on the other. Tag values are stored in memory (currently at the top of DRAM
* and there is a cache of 32ki tags (representing 1MB memory) stored in BRAM.
* Read responses are amended with the correct tag value and write requests update
* the value in the tag cache (which is later written back to memory).
*
*****************************************************************************/

Interface TagControllerIfc

Two sub-interfaces:

  • interface Slave#(CheriMemRequest, CheriMemResponse) cache;
    • // a server interface
    • // connected to l2 cache’s memory;
    • // as Slave means it receives request from l2cache’s memory, and generate response for the l2Cache’s memory interface to get from.
    • // does this mean TagController is under l2-cache? Yes. See 2017 paper.
  • interface Master#(CheriMemRequest, CheriMemResponse) memory;
    • // a client interface
    • // provide the proxy interface for memory access to physical memory. i.e., this is the CPU’s memory interface that will be connected to the main memory.
    • // as master means it send request to memory and grab response back by calling interface.
    • // The memory interface is slave in this case: phycial memory interface?

The definition of CheriMemRequest/CheriMemResponse is in MemTypes.bsv, which contains the data to be write/read with tags (Data#).

Module mkTagController(TagControllerIfc)

Input/Output

The connections:

mkConnection(l2CacheMemory, tagController.cache);

mkConnection(tagLookup.cache.response,toCheckedPut(ff2fifof(lookupRsp)));

     ...
-----------------------
| (slave/srv/cache)   |
|  L2Cache            |  
| (master/clt/memory) |
-----------------------
     /\
     |
     \/  
-------------------------------------------------------------------
| (slave/srv/cache) |                                             |
|                   |                                             |
|                   |                                             |
|   TagController   |   -> tag cache req: tagLookup..put(req)     |
|         |         |   <- tag cache hit: lookupRsp..first()      |
|   mem   |   tag <------------------>  (slave/srv/cache)         |
|    |    |                                        |              |
|    |    |                                 tagLookup (~L3Cache)  |
|    |    |                                        |              |
|    |    |     |<--------------------> (master/clt/memory)       |
|    \/   |     \/         <- tag cache miss (forwardLookupReqs)  |
| (master/clt/memory)      -> tag mem rsps ()                     |
-------------------------------------------------------------------
         /\
         |                  
         \/
| -----------------------------------------------------------------
|  mReqs.enq(r)  mRsps.first                                      |
|          memory                                                 |
-------------------------------------------------------------------

Input from L2Cache:

Overview: Get input request, hand it over to fifo mReqs and tagLookup.cache.request, and wait for the response

  • memory request CheriMemRequest req: -> mReqs.enq(req)

    • input from L2Cache’s master interface memory.request.get() which returns memReqs.first from L2Cache:
    • received via TagController’s slave interface cache.put() which enqueue the request mReqs.enq(req), which will

    • The cache.put() will also put the request in tagLookup module: tagLookup.cache.request.put(req), which will

      // cherilibs/trunk/TagController.bsv

interface Slave cache;
// request side
///////////////////////////////////////////////////////
interface CheckedPut request;
method Bool canPut() = slvCanPut;
method Action put(CheriMemRequest req) if (slvCanPut);
debug2("tagcontroller", $display("<time %0t TagController> New request: ", $time, fshow(req)));
mReqs.enq(req);
`ifndef NOTAG
tagLookup.cache.request.put(req);
`endif
endmethod
endinterface

Output to L2Cache

Output2: (Slave) memory response from here up to l2.

Call peekMemResponse() when slvCanGet, means there is a response.

  • mRsps.first/.deq() , lookupRsp.first()/.deq()

  • response will be returned as CheriMemResponse to L2Cache

    // cherilibs/trunk/TagController.bsv

interface Slave cache;
// response side
///////////////////////////////////////////////////////
interface CheckedGet response;
method Bool canGet() = slvCanGet;
method CheriMemResponse peek() = peekMemResponse();
method ActionValue#(CheriMemResponse) get() if (slvCanGet);
  // prepare memory response
  CheriMemResponse resp = peekMemResponse();
  // dequeue memory response fifo
  mRsps.deq();
  `ifndef NOTAG
  // in case of read response ...
  if (resp.operation matches tagged Read .rop) begin
    // on the last flit,
    if (rop.last) begin
      lookupRsp.deq(); // dequeue the tag lookup response fifo
      frame <= 0;  // reset the current frame
    end else frame <= frame + 1; // for non last flits, increment frame
  end
  `endif
  debug2("tagcontroller", $display("<time %0t TagController> Returning response: ", $time, fshow(resp)));
  return resp;
endmethod
endinterface
endinterface

Output/Input to/from Memory

(Master/Client) memory requests CheriMemRequest req from tagController down to mem;

If got a response from memory, the request will be distinguished according to its reqType:

  • if tag request, will send response to tag cache tagLookup.memory.response

  • otherwise, will send response to data response fifo mRsps which will be forwarded to l2Cache, as in above section.

    // cherilibs/trunk/TagController.bsv

interface Master memory;

// an interface of type CheckedGet#(CheriMemRequest mReqs, see [cherilibs/trunk/MasterSlave.bsv](../master-slave)
// used to send request to memory
interface request  = toCheckedGet(ff2fifof(mReqs));

// an interface of type CheckedPut
// used to get response from memory
interface CheckedPut response;
method Bool canPut();
  return (mRsps.notFull() && tagLookup.memory.response.canPut());
endmethod
method Action put(CheriMemResponse r) if (mRsps.notFull() && tagLookup.memory.response.canPut());
  `ifdef NOTAG
    mRsps.enq(r);
  `else
    MemReqType reqType = (r.masterID == mID) ? TagLookupReq : StdReq;
    debug2("tagcontroller", $display("<time %0t TagController> response from memory: source=%x ", $time, reqType, fshow(r)));
    if (reqType == TagLookupReq) begin
      if (r.operation matches tagged Read.rop) tagLookup.memory.response.put(r);
      debug2("tagcontroller", $display("<time %0t TagController> tag response", $time));
    end else begin
      mRsps.enq(r);
      debug2("tagcontroller", $display("<time %0t TagController> memory response", $time));
    end
  `endif
endmethod
endinterface
endinterface

States:

TagLookupIfc tagLookup <-mkMultiLevelTagLookup(…);

  • CheriMasterID mID = 12; // masterID used for memory requests from the lookup engine

  • TagTableStructure; // declares and initialize the tableStructure Vector of interger

  • FF#(CheriTagResponse,4) lookupRsp; // lookup responses fifo

  • Reg#(Bit#(2)) frame; // lookup response frame to access (for multi-flit transaction)

  • FF#(CheriMemRequest, 2) mReqs <- mkFF(); // memory request fifo

  • FF#(CheriMemResponse, 32) mRsps <- mkUGFF(); // memory response fifo

submodule tagLookup

tagLookup module is the core module that handles tag search and tag cache miss, where a cache miss will be forwarded as a normal memory request via mReqs in this module.

A memory request is sent to tagLookup via connections on Slave request interface

A memory response can be get from tagLookup via this connection: Connetion between tagLookup.cache.response(A Slave#/Server# CheckedGet#) – lookupRsp (toCheckedPut#):

mkConnection(tagLookup.cache.response,toCheckedPut(ff2fifof(lookupRsp)));

A tag lookup miss will be forward as a regular memory request in rule forwardLookupReqs

Rules

  • rule forwardLookupReqs(tagLookup.memory.request.canGet() && mReqs.notFull()); // forwards the tag lookup requests to the memory interface

  • rule debug; // print debug2 information every cycle?

Rule forwardLookupReqs

Forwards tag lookup requests (tagLookup.memory.request) to the memory interface (via fifo mReqs.enq(r)).

Note: if tagLookup module got a tag cache miss, then this rule will be triggered and new tag-related memory request will be put in the queue mReqs.

  // forwards tag lookup requests to the memory interface
  rule forwardLookupReqs(tagLookup.memory.request.canGet() && mReqs.notFull());
    debug2("tagcontroller",
      $display(
        "<time %0t TagController> Injecting request from tag lookup engine: ",
        $time, fshow(tagLookup.memory.request.peek())
    ));
    CheriMemRequest r <- tagLookup.memory.request.get();
    mReqs.enq(r);
  endrule

helper Function:

peekMemResponse(): grab a response from fifo mRsps, parse the tag if read and return the response struct.

  • function CheriMemResponse peekMemResponse(); // look at the next memory response

    • prepare mRsps.first for return
    • if read, need to construct tags, i.e. parse the tag response from tag cache and send response with tag:
    • Vector#(TDiv#(CheriDataWidth,CapWidth),Bool) tags = replicate(True); ??? Now it seems (CheriDataWidth == CapWidth), and tags is 1-bit long?
    • look at the tag lookup response: case lookupRsp.first(): // check this place if we will expand tag bits.
    • if matches Covered .ts: tags = unpack(ts[frame]);

    • if matches Unconvered : tags = unpack(0);

      // cherilibs/trunk/TagController.bsv

// look at the next memory response
function CheriMemResponse peekMemResponse();
// look at the next response from the memory master interface
CheriMemResponse resp    = mRsps.first;
// initialise new response to response coming from memory
CheriMemResponse newResp = resp;
// in case of read, need to construct the tags
if (resp.operation matches tagged Read .rop) begin
Vector#(TDiv#(CheriDataWidth,CapWidth),Bool) tags = replicate(True);
`ifndef NOTAG
// look at the tag lookup response
case (lookupRsp.first()) matches
tagged Covered .ts : tags = unpack(ts[frame]); // check this place if we will expand tag bits.
tagged Uncovered   : tags = unpack(0);
endcase
`endif
// update the new response with appropriate tags
newResp.data.cap = tags;
end
return newResp;
endfunction

Steps:

  • receive request from l2cache and put them in 2 fifo: one mReqs, another at tagLookup.cache.request
  • ask memory to process mReqs;
  • ask tagLookup to find tags in tag cache;
  • response to l2Cache based on responses from memory (mRsps) and tag cache (lookupRsp)

Slave - cache

mkConnection(l2CacheMemory, tagController.cache);

two sub-interfaces that are connected with the L2CacheMemory:

  • request side interface: CheckedPut request; // to receive mem req from L2Cache
  • response side interface: CheckedGet response; // to send response back to L2Cache

defined in

// cherilibs/trunk/MasterSlave.bsv

interface Slave#(type req_t, type rsp_t);
    interface CheckedPut#(req_t) request;
    interface CheckedGet#(rsp_t) response;
endinterface

The interface declaration of CheckedPut/Get, similar to Get/Put in Bluespec, where Get is for others to get sth out, and Put for others to put sth in; CheckedPut/Get are defined in cherilibs/trunk/MasterSlave.bsv

Slave CheckedPut request

Type of CheckedPut: is for L2Cache to send in new request.

methods:

  • method Bool canPut() = slvCanPut;
  • method Action put(CheriMemRequest req) if (slvCanPut);
    • put req to queue mReqs.enq(req);
    • put req to tagLookup.cache.request.put(req);

Slave CheckedGet response

Type of CheckedGet: is for L2Cache to

methods:

  • method Bool canGet() = slvCanGet;
  • method CheriMemResponse peek() = peekMemResponse();
  • method ActionValue#(CheriMemResponse) get() if (slvCanGet);
    • peek mem response and save as resp: CheriMemResponse resp = peekMemResponse();
    • dequeue memory response fifo: mRsps.deq()

Master - memory

Connected to main memory interface:

MIPSTop.bsv (mkMIPSMemory theMem: interface memory = theMem.memory) ==>

Memory.bsv: (MIPSMemory Ifc: Master# memory = topMemIfc = tagControllerMemory) ==> this master interface

The memory interface transfers bit-convertable data type of CheriMemRequest and CheriMemResponse.

No distinguish to the request sent to the memory, but distinguish the response being send back from the memory. The bus will contain these bits being transferred to and from the memory.

Two sub-interfaces

  • interface CheckedGet request
  • interface CheckedPut response

declared in:

// cherilibs/trunk/MasterSlave.v

interface Master#(type req_t, type rsp_t);
    interface CheckedGet#(req_t) request;
    interface CheckedPut#(rsp_t) response;
endinterface

Master CheckedGet request

request is an interface of CheckedGet#()

interface request = toCheckedGet(ff2fifof(mReqs));

Definition of toCheckedGet(fifo) (also see MemTypes)

// cherilibs/trunk/MasterSlave.bsv

instance ToCheckedGet#(FIFOF#(data_t), data_t);
    function CheckedGet#(data_t) toCheckedGet (FIFOF#(data_t) f) =
        interface CheckedGet#(data_t);
            method canGet = f.notEmpty;
            method data_t peek if (f.notEmpty);
              return f.first;
            endmethod
            method ActionValue#(data_t) get if (f.notEmpty);
              f.deq; 
              return f.first;
            endmethod
        endinterface;
endinstance

Master CheckedPut response

Used to grab response from main memory. Responses of main memory will be found at fifo mRsps.

// cherilibs/trunk/TagController.bsv

interface CheckedPut response;
      method Bool canPut();
        return (mRsps.notFull() && tagLookup.memory.response.canPut());
      endmethod
      method Action put(CheriMemResponse r) if (mRsps.notFull() && tagLookup.memory.response.canPut());
        `ifdef NOTAG
          mRsps.enq(r);
        `else
          MemReqType reqType = (r.masterID == mID) ? TagLookupReq : StdReq;
          debug2("tagcontroller", $display("<time %0t TagController> response from memory: source=%x ", $time, reqType, fshow(r)));
          if (reqType == TagLookupReq) begin
            if (r.operation matches tagged Read.rop) tagLookup.memory.response.put(r);
            debug2("tagcontroller", $display("<time %0t TagController> tag response", $time));
          end else begin
            mRsps.enq(r);
            debug2("tagcontroller", $display("<time %0t TagController> memory response", $time));
          end
        `endif
      endmethod
    endinterface
Created May 3, 2020 // Last Updated May 23, 2020

If you could revise
the fundmental principles of
computer system design
to improve security...

... what would you change?