AEM 6430 - AEM Asset Share Commons Presigned S3 Download URLs for Downloading GB files

Goal

Add a Search Results component /apps/eaem-asc-s3-presigned-urls/components/results extending Asset Share Commons Search Results component /apps/asset-share-commons/components/search/results to download the assets from Amazon S3 (AEM is configured with S3 Data Store)

Default download uses the Asset Download Servlet - /content/dam.assetdownload.zip/assets.zip, however if the need is to offload asset downloading from AEM to a specialized service like S3, the following solution can be tried out... Downloading large files (> 1GB) directly from S3 storage can help improve the performance of AEM publish instances...

In the following solution, adding selector extension eaems3download.html to any asset path e.g. /content/dam/experience-aem/123.jpg.eaems3download.html creates a redirect (302) response with Presigned S3 url

If AEM is on Oak 1.10 or later, Direct Binary Access can also improve AEM performance by downloading binaries directly from storage provider (S3, Azure etc)

For creating S3 download links for large carts check this post

Package Install | Github


OSGI Configuration

                  Copy the S3 Data Store bucket name from - http://localhost:4502/system/console/configMgr/org.apache.jackrabbit.oak.plugins.blob.datastore.SharedS3DataStore

                  Add the S3 Datastore bucket name in - http://localhost:4502/system/console/configMgr/apps.experienceaem.assets.EAEMS3Service

Component Configuration

Component Rendering

Solution

1) Assuming AEM is running on a AWS EC2 instance configured with a IAM role for accessing S3, you can use the following statement to create S3 client

                   AmazonS3 s3Client = AmazonS3ClientBuilder.defaultClient();

if using accessKey and secretKey the client can be created using...

                   AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
                             .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials("accessKey", "secretKey")))
                             .build();

2)  Add a service for reading the bucket configuration and creating Presigned urls apps.experienceaem.assets.EAEMS3Service. AEM creates the SHA256 of asset binary, uses it as S3 object id and stores the id in segment store (not available via asset metadata) before uploading the binary to S3. getS3AssetIdFromReference() function gets the s3 object id from binary reference...

package apps.experienceaem.assets;

import com.amazonaws.HttpMethod;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.GeneratePresignedUrlRequest;
import com.amazonaws.services.s3.model.ResponseHeaderOverrides;
import com.day.cq.commons.jcr.JcrConstants;
import com.day.cq.dam.api.Asset;
import com.day.cq.dam.commons.util.DamUtil;
import org.apache.commons.lang3.StringUtils;
import org.apache.jackrabbit.api.JackrabbitValue;
import org.apache.jackrabbit.api.ReferenceBinary;
import org.apache.sling.api.resource.Resource;
import org.osgi.service.component.annotations.Activate;
import org.osgi.service.component.annotations.Component;
import org.osgi.service.metatype.annotations.AttributeDefinition;
import org.osgi.service.metatype.annotations.AttributeType;
import org.osgi.service.metatype.annotations.Designate;
import org.osgi.service.metatype.annotations.ObjectClassDefinition;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.jcr.Node;
import javax.jcr.Property;
import javax.jcr.Value;
import java.net.URL;
import java.util.Date;

@Component(
        immediate=true ,
        service={ EAEMS3Service.class }
)
@Designate(ocd = EAEMS3Service.Configuration.class)
public class EAEMS3Service {
    private final Logger logger = LoggerFactory.getLogger(this.getClass());

    private static AmazonS3 s3Client = AmazonS3ClientBuilder.defaultClient();

    private long singleFileS3Expiration = (1000 * 60 * 60);
    private String s3BucketName = "";

    @Activate
    protected void activate(EAEMS3Service.Configuration configuration) {
        singleFileS3Expiration = configuration.singleFileS3Expiration();
        s3BucketName = configuration.s3BucketName();
    }

    public String getS3PresignedUrl(Resource resource){
        String presignedUrl = "";

        if( (resource == null) || !DamUtil.isAsset(resource)){
            logger.warn("Resource null or not a dam:Asset");
            return presignedUrl;
        }

        if(StringUtils.isEmpty(s3BucketName)){
            logger.warn("S3 Bucket Name not configured");
            return presignedUrl;
        }

        Asset s3Asset = DamUtil.resolveToAsset(resource);

        if(s3Asset == null){
            return presignedUrl;
        }

        try{
            String objectKey = getS3AssetIdFromReference(resource);

            if(StringUtils.isEmpty(objectKey)){
                logger.debug("S3 object key empty, could be in segment store - " + resource.getPath());

                presignedUrl = resource.getPath();

                return presignedUrl;
            }

            logger.debug("Path = " + resource.getPath() + ", S3 object key = " + objectKey);

            if(StringUtils.isEmpty(objectKey)){
                return presignedUrl;
            }

            ResponseHeaderOverrides nameHeader = new ResponseHeaderOverrides();
            nameHeader.setContentType(s3Asset.getMimeType());
            nameHeader.setContentDisposition("attachment; filename=" + resource.getName());

            GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(s3BucketName, objectKey)
                    .withMethod(HttpMethod.GET)
                    .withResponseHeaders(nameHeader)
                    .withExpiration(getSingleFileS3ExpirationDate());

            URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);

            presignedUrl = url.toString();

            logger.debug("Path = " + resource.getPath() + ", S3 presigned url = " + presignedUrl);
        }catch(Exception e){
            logger.error("Error generating s3 presigned url for " + resource.getPath(), e);
            presignedUrl = resource.getPath();
        }

        return presignedUrl;
    }

    public Date getSingleFileS3ExpirationDate(){
        Date expiration = new Date();

        long expTimeMillis = expiration.getTime();
        expTimeMillis = expTimeMillis + singleFileS3Expiration;

        expiration.setTime(expTimeMillis);

        return expiration;
    }

    public static String getS3AssetIdFromReference(final Resource assetResource) throws Exception {
        String s3AssetId = StringUtils.EMPTY;

        if( (assetResource == null) || !DamUtil.isAsset(assetResource)){
            return s3AssetId;
        }

        Resource original = assetResource.getChild(JcrConstants.JCR_CONTENT + "/renditions/original/jcr:content");

        if(original == null) {
            return s3AssetId;
        }

        Node orgNode = original.adaptTo(Node.class);

        if(!orgNode.hasProperty("jcr:data")){
            return s3AssetId;
        }

        Property prop = orgNode.getProperty("jcr:data");

        ReferenceBinary value = (ReferenceBinary)prop.getBinary();

        s3AssetId = value.getReference();

        if(StringUtils.isEmpty(s3AssetId) || !s3AssetId.contains(":")){
            return s3AssetId;
        }

        s3AssetId = s3AssetId.substring(0, s3AssetId.lastIndexOf(":"));

        s3AssetId = s3AssetId.substring(0, 4) + "-" + s3AssetId.substring(4);

        return s3AssetId;
    }

    public static String getS3AssetId(final Resource assetResource) {
        String s3AssetId = StringUtils.EMPTY;

        if( (assetResource == null) || !DamUtil.isAsset(assetResource)){
            return s3AssetId;
        }

        Resource original = assetResource.getChild(JcrConstants.JCR_CONTENT + "/renditions/original");

        if(original == null) {
            return s3AssetId;
        }

        //performance hit when the file size cross several MBs, GBs
        Value value = (Value)original.getValueMap().get(JcrConstants.JCR_CONTENT + "/" + JcrConstants.JCR_DATA, Value.class);

        if (value != null && (value instanceof JackrabbitValue)) {
            s3AssetId = gets3ObjectIdFromJackrabbitValue((JackrabbitValue) value);
        }

        return s3AssetId;
    }

    private static String gets3ObjectIdFromJackrabbitValue(JackrabbitValue jrValue) {
        if (jrValue == null) {
            return StringUtils.EMPTY;
        }

        String contentIdentity = jrValue.getContentIdentity();

        if (StringUtils.isBlank(contentIdentity)) {
            return StringUtils.EMPTY;
        }

        int end = contentIdentity.lastIndexOf('#');

        contentIdentity = contentIdentity.substring(0, end != -1 ? end : contentIdentity.length());

        return contentIdentity.substring(0, 4) + "-" + contentIdentity.substring(4);
    }

    @ObjectClassDefinition(
            name = "Experience AEM S3 for Download",
            description = "Experience AEM S3 Presigned URLs for Download"
    )
    public @interface Configuration {

        @AttributeDefinition(
                name = "Single file download S3 URL expiration",
                description = "Single file download Presigned S3 URL expiration",
                type = AttributeType.LONG
        )
        long singleFileS3Expiration() default (1000 * 60 * 60);

        @AttributeDefinition(
                name = "S3 Bucket Name e.g. eaem-s3-bucket",
                description = "S3 Bucket Name e.g. eaem-s3-bucket",
                type = AttributeType.STRING
        )
        String s3BucketName();
    }
}

3) Add a servlet apps.experienceaem.assets.EAEMS3DownloadServlet for creating redirect response (302) with presigned urls...

package apps.experienceaem.assets;

import org.apache.sling.api.SlingHttpServletRequest;
import org.apache.sling.api.SlingHttpServletResponse;
import org.apache.sling.api.servlets.SlingAllMethodsServlet;
import org.osgi.service.component.annotations.Component;
import org.osgi.service.component.annotations.Reference;

import javax.servlet.Servlet;
import javax.servlet.ServletException;
import java.io.IOException;

@Component(
        service = Servlet.class,
        property = {
                "sling.servlet.methods=GET",
                "sling.servlet.resourceTypes=dam:Asset",
                "sling.servlet.selectors=eaems3download",
                "sling.servlet.extensions=html"
        }
)
public class EAEMS3DownloadServlet extends SlingAllMethodsServlet {

    @Reference
    private EAEMS3Service eaems3Service;

    public final void doGet(SlingHttpServletRequest request, SlingHttpServletResponse response)
                            throws ServletException, IOException {
        response.sendRedirect(eaems3Service.getS3PresignedUrl(request.getResource()));
    }
}

4) Create component /apps/eaem-asc-s3-presigned-urls/components/results with sling:resourceSuperType /apps/asset-share-commons/components/search/results

5) Create the dialog /apps/eaem-asc-s3-presigned-urls/components/results/cq:dialog with following code, for adding eaemUseS3 field

<?xml version="1.0" encoding="UTF-8"?>
<jcr:root xmlns:sling="http://sling.apache.org/jcr/sling/1.0" xmlns:cq="http://www.day.com/jcr/cq/1.0" xmlns:jcr="http://www.jcp.org/jcr/1.0" xmlns:nt="http://www.jcp.org/jcr/nt/1.0"
    jcr:primaryType="nt:unstructured"
    jcr:title="Date Range Filter"
    sling:resourceType="cq/gui/components/authoring/dialog"
    extraClientlibs="[core.wcm.components.form.options.v1.editor,asset-share-commons.author.dialog]">
    <content
        jcr:primaryType="nt:unstructured"
        sling:resourceType="granite/ui/components/coral/foundation/container">
        <items jcr:primaryType="nt:unstructured">
            <tabs jcr:primaryType="nt:unstructured">
                <items jcr:primaryType="nt:unstructured">
                    <tab-2 jcr:primaryType="nt:unstructured">
                        <items jcr:primaryType="nt:unstructured">
                            <column jcr:primaryType="nt:unstructured">
                                <items jcr:primaryType="nt:unstructured">
                                    <card
                                        jcr:primaryType="nt:unstructured"
                                        sling:resourceType="granite/ui/components/coral/foundation/form/select"
                                        emptyText="Choose an render for Card results"
                                        extensionTypes="[/apps/eaem-asc-s3-presigned-urls/components/results/result/card]"
                                        fieldDescription="Resource type used to render card results"
                                        fieldLabel="Card Result Renderer"
                                        name="./cardResultResourceType">
                                        <datasource
                                            jcr:primaryType="nt:unstructured"
                                            sling:resourceType="asset-share-commons/data-sources/result-resource-types"/>
                                    </card>
                                    <use-s3
                                        jcr:primaryType="nt:unstructured"
                                        sling:orderBefore="name"
                                        sling:resourceType="granite/ui/components/coral/foundation/form/checkbox"
                                        fieldDescription="Use AWS S3 Presigned URLs for Download"
                                        name="./eaemUseS3"
                                        text="S3 Presigned urls for download"
                                        value="{Boolean}true"/>
                                </items>
                            </column>
                        </items>
                    </tab-2>
                </items>
            </tabs>
        </items>
    </content>
</jcr:root>

6) In /apps/eaem-asc-s3-presigned-urls/components/results/result/card/templates/card.html use the following code to add S3DOWNLOAD in card view

<li data-sly-test="${config.downloadEnabled}">
 <div data-sly-unwrap data-sly-test="${!properties.eaemUseS3}">
  <button class="ui link button"
    data-asset-share-id="download-asset"
    data-asset-share-asset="${asset.path}"
    data-asset-share-license="${config.licenseEnabled ? asset.properties['license'] : ''}">${'Download' @ i18n}</button>
 </div>
 <div data-sly-unwrap data-sly-test="${properties.eaemUseS3}">
  <a href="${asset.path}.eaems3download.html" target="_blank">S3DOWNLOAD</a>
 </div>
</li>