Correct way of uploading from buffer?

Comments

21 comments

  • Avatar
    Aleksandar Kostadinov

    Hi,

    Thanks for your message.

    You can indeed upload from a buffer and the way to do this is via the upload_stream method. You'll first need to turn the buffer into a readable stream, which I'm doing using the streamifier library in my example. I'm also setting the file to be uploaded into the "foo" folder, but of course, you can add any other upload parameters to that object that you need.

    let cloudinary = require("cloudinary").v2;
    let streamifier = require('streamifier');

    let cld_upload_stream = cloudinary.uploader.upload_stream(
    {
    folder: "foo"
    },
    function(error, result) {
    console.log(error, result);
    }
    );

    streamifier.createReadStream(req.file.buffer).pipe(cld_upload_stream);

    May I please ask you to try this and let me know how it goes.

    Best regards,

    Aleksandar

    2
    Comment actions Permalink
  • Avatar
    Arquibusco

    Thank you very much for your reply Aleksandar, finally someone helps. I have tried this function before, but not with pipe method. I have two issues with it this way:

    cloudinary.v2.uploader.upload_stream({ format: 'jpg' }, (err, res) => {
            if (err) {
              console.log(err);
            } else {
              console.log(`Upload succeed: ${res}`);
              // filteredBody.photo = result.url;
            }
          }).end(req.file.processedImage);

    The problems with it is that when I try to add the url to the photo so I can store in DB, it returns a blank result. So I tried using async function and await the result of the upload_stream, but is seems to not return a promise, even though it says it returns a promise. 

    I will now try your solution, but do you perhaps know if this function returns a promise or when can i find more about it (seems to not be in the docs)?

    0
    Comment actions Permalink
  • Avatar
    Aleksandar Kostadinov

    It should indeed work with Promises too. Perhaps you can also try something along the lines of:

    let cloudinary = require("cloudinary").v2;
    let streamifier = require('streamifier');

    let uploadFromBuffer = (req) => {

       return new Promise((resolve, reject) => {

         let cld_upload_stream = cloudinary.v2.uploader.upload_stream(
          {
            folder: "foo"
          },
          (error: any, result: any) => {

            if (result) {
             resolve(result);
           } else {
              reject(error);
             }
           }
         );

         streamifier.createReadStream(req.file.buffer).pipe(cld_upload_stream);
       });

    };

    let result = await uploadFromBuffer(req);

    I'll take a look internally whether we have notes on why it isn't present in the main documentation. On a similar note, the callback approach is mentioned in the ReadMe section of the repository - https://github.com/cloudinary/cloudinary_npm#cloudinaryupload_stream

    4
    Comment actions Permalink
  • Avatar
    Arquibusco

    I really want to say thank you for your efforts on helping me! I really appreciate it. 

    1. What I meant with return promises is: if I await

    cloudinary.uploader.upload

    I obtain very different results than if I await 

    cloudinary.uploader.upload_stream

    awaiting uploader.upload, returns the successful upload info, like url, public_id,etc.., meanwhile with upload_stream it doesn't. 

    After your response I understood that maybe I wasn't passing the right parameters.

    2. I know maybe this is a very noob question, but is buffering an image the right way to go? I mean, I want for users to upload an image and then send it to cloudinary, without it being in the server, but as i've read to few posts on buffering images in Cloudinary, it makes me think if i'm on track.

    3. When images are buffered to cloudinary, does this take up server memory or the clients browser memory? Because maybe buffering images can be more harmful than saving to server and then deleting with the file system.

    I hope i'm not being a pain to you with so many questions, but I really like Cloudinary and want to make things work well.

    0
    Comment actions Permalink
  • Avatar
    Raz Ziv

    Hi,

    Based on your last comment, I would just like to confirm that what you mean when saying "After your response, I understood that maybe I wasn't passing the right parameters." is that you've managed to resolve the problem with the missing response in the Promise of the upload_stream() method, by fixing an issue you had with the parameters. Is that correct?

    To answer your second question, based on what you are trying to achieve, I would say the best approach will be embedding our Upload Widget in your site's code, so your clients will upload their media directly from their browser to Cloudinary servers.

    With the buffer stream approach, the file indeed isn't saved on your server, but it does go through your server's memory, meaning it is still uploaded from your server and not from the client-side. 
    To avoid any usage of your server's resources, the way to go is to either use the Upload Widget mentioned above or a direct upload from the browser using our jQuery library, which its upload functionality is based on blueimp's jQeury file uploader.

    I hope this answers all of your questions :)

    0
    Comment actions Permalink
  • Avatar
    Arquibusco

    Hi,

    What is working for me is:

    //converting buffer to usable format

    const imageFile=dataUri.format(
    req.file.processedImage.info.format,
    req.file.processedImage.data
    );
     
    const result = await cloudinary.uploader.upload(imageFile.content, {
    public_id: req.file.filename,
    folder: "profile/"
    });
     
    //using the result to save to database
    0
    Comment actions Permalink
  • Avatar
    Raz Ziv

    Let's try and take a step back and use the upload_stream() method to upload a locally stored image with a promise.

    Please try and use the code below to see if you get the response printed to your console once the promise has been fulfilled:

    var fs = require("fs");

    let streamUpload = (file_name) => {

      return new Promise((resolve, reject) => {

        let stream = cloudinary.uploader.upload_stream(

          (error, result) => {

            if (result) {

              resolve(result);

            } else {

              reject(error);

            }

          }

        );

        fs.createReadStream(file_name).pipe(stream);
     
    });

    };

    async function async_func(file_name) {

      let result = await streamUpload(file_name);

      console.log(result);
    }

    let file_name = __dirname + '/my_picture.jpg';

    async_func(file_name);

    Just please make sure that you first temporarily add a photo to the same directory as the script with the name of 'my_picture.jpg' or alternatively change the path of the file_name variable.

    This has been tested on my machine and the response was received similarly to the response from the regular upload() method.

    Once we ensure this works for you as expected, we will move on to replace the local file upload with a buffer image.

    Best,
    Raz

    1
    Comment actions Permalink
  • Avatar
    Arquibusco

    Hi Raz, 

    I'll keep this short to make it easier to continue:

    Yes it worked! 

    0
    Comment actions Permalink
  • Avatar
    Raz Ziv

    Excellent, thank you for the update, I'm glad to hear that.

    So now let's proceed with installing Streamifier via npm, as Aleksander previously suggested.

    Once installed, kindly try and change the upload of the local file to upload a buffer-converted string:

    First, add:

    let streamifier = require('streamifier');

    And then make sure you keep the same code but only replace the local file with the actual buffer request.

    let streamUpload = (req) => {

    return new Promise((resolve, reject) => {

    let stream = cloudinary.uploader.upload_stream(

    (error, result) => {

    if (result) {

    resolve(result);

    } else {

    reject(error);

    }

    }

    );

    fs.createReadStream(req.file.buffer).pipe(stream);

    });

    };

    async function async_func(req) {

    let result = await streamUpload(req);

    console.log(result);

    }

    async_func(req);

    In case you are still experiencing any issues with getting back the upload response when running the above, please open a support ticket at support@cloudinary.com and one of our agents will be happy to further help you with getting this issue resolved.

    Best,
    Raz

    3
    Comment actions Permalink
  • Avatar
    Arquibusco

    Thank you all for your help! Outstanding service.

    0
    Comment actions Permalink
  • Avatar
    Raz Ziv

    You're most welcome! Happy that we could help 🙂

    0
    Comment actions Permalink
  • Avatar
    Neckaros

    Great support indeed.

    do you plan to return a promise from:

    cloudinary.uploader.upload_stream

    to simplify code?

    0
    Comment actions Permalink
  • Avatar
    Raz Ziv

    Hey Neckaros,

    Thanks for your feedback :)

    We do intend to return a promise from the upload_stream() and some other methods as well. This is expected to take place in one of our near-future releases, however, we currently don't have an ETA to share yet.

    Thank you for your patience and understanding 🙏🏻.

    Best,
    Raz

    1
    Comment actions Permalink
  • Avatar
    olohundare nasirudeen

    To be able to use async/await with cloudinary, you can do something like this

     

    const cloudinary = require("cloudinary").v2;

    const { promisify } = require("util");

    const cloudinaryUploadLarge = promisify(cloudinary.uploader.upload_large).bind(

      cloudinary

    );


    let response = await cloudinaryUploadLarge(name, {

        resource_type: "raw",

        public_id: your public id,

      });




    console.log(response)
    0
    Comment actions Permalink
  • Avatar
    abdel

    I found out that the solution of passing the 2 parameters (error, result) doesn't work anymore cause now it returns the error on the response so i changed this part like this:

    let stream = cloudinary.uploader.upload_stream(

    (result) => {

    if (!result.error) {

    resolve(result);

    } else {

    reject(result.error);

    }

    }

    );
    0
    Comment actions Permalink
  • Avatar
    Daniel Mendoza

    @Aspiiire YOUTUBE

    The callback has the first parameter as error and the second as result. This is expected behavior.

    let stream = cloudinary.uploader.upload_stream(

    (error, result) => {
    if (result) {
    resolve(result);
    } else {
    reject(error);
    }
    }
    );
    0
    Comment actions Permalink
  • Avatar
    Shameek Agarwal

    in cloudinary.config.ts

    import { v2 as cloudinary } from 'cloudinary';
    cloudinary.config({
    cloud_name: process.env.CLOUDINARY_CLOUD_NAME,
    api_key: process.env.CLOUDINARY_API_KEY,
    api_secret: process.env.CLOUDINARY_API_SECRET,
    });
    export { cloudinary };
    function to upload image using buffer
    import { UploadApiResponse } from 'cloudinary';
    import streamifier from 'streamifier';
    import { cloudinary } from './config/cloudinary.config';

    async uploadImage(buffer: Buffer): Promise<UploadApiResponse> {
    return new Promise((resolve, reject) => {
    const uploadStream = cloudinary.uploader.upload_stream(
    {
    folder: 'events',
    upload_preset: 'ml_default',
    },
    (error: Error, result: UploadApiResponse) => {
    if (result) resolve(result);
    else reject(error);
    },
    );
    streamifier.createReadStream(buffer).pipe(uploadStream);
    });
    }

     

    1
    Comment actions Permalink
  • Avatar
    Adam Herbert

    I am hitting a file too large error when uploading a video via a stream. How can you increase the upload size over 100mb? Do you just specify the chunk size or do you have to use something like the upload_large method?

    0
    Comment actions Permalink
  • Avatar
    Danny Valentine

    Hi Adam.

    Thanks for getting in touch.

    This is actually an account limitation. The maximum file size for video is in fact 100MB on our free plan, however if you contact us via support@cloudinary.com we may be able to come up with a solution for you.

    Thanks,
    -Danny

    0
    Comment actions Permalink
  • Avatar
    Norbert Benkó

    And could you suggest a way how to do it with a larger stream (larger than 100mb) in chunks?

    0
    Comment actions Permalink
  • Avatar
    Stephen Doyle

    Hi Norbert,

    If you're using Cloudinary's Node SDK as the other commenters on this thread were, there's a dedicated method in the SDK for that, named `upload_large_stream`

    If you're implementing the upload yourself, there's a guide here with the specific requirements for how the chunks should be structured: https://support.cloudinary.com/hc/en-us/articles/208263735-Guidelines-for-self-implementing-chunked-upload-to-Cloudinary

    Thanks,
    Stephen

    0
    Comment actions Permalink

Please sign in to leave a comment.